[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:37:56 dist-test-slave-f7th krb5kdc[31998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:37:56 dist-test-slave-f7th krb5kdc[31998](info): set up 2 sockets
Aug 12 16:37:56 dist-test-slave-f7th krb5kdc[31998](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 16:37:57.786329 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:37:57 dist-test-slave-f7th krb5kdc[31998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:37:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:37:57.977337 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43495
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:43495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:37:58.484230 32012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:37:58.484756 32012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:37:58.485003 32012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:37:58.493664 32012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:37:58.493794 32012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:37:58.493868 32012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:37:58.493964 32012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:37:58.494029 32012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:37:58.506443 32012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:43495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:37:58.508558 32012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:37:58.510672 32012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:37:58.530321 32018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:37:58.530326 32019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:37:58.530169 32021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:37:58.534142 32012 server_base.cc:1034] running on GCE node
I20260812 16:37:58.535465 32012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:37:58.538017 32012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:37:58.539254 32012 hybrid_clock.cc:648] HybridClock initialized: now 1786552678539140 us; error 102 us; skew 500 ppm
I20260812 16:37:58.539835 32012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:37:58 dist-test-slave-f7th krb5kdc[31998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552678, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:37:58.544044 32012 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:37:58.546645 32012 webserver.cc:460] Webserver started at http://127.31.61.190:33855/ using document root <none> and password file <none>
I20260812 16:37:58.548039 32012 fs_manager.cc:359] Metadata directory not provided
I20260812 16:37:58.548197 32012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:37:58.548743 32012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:37:58.553382 32012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "588f801e5007480bbfd2c6860aeb81a2"
format_stamp: "Formatted at 2026-08-12 16:37:58 on dist-test-slave-f7th"
server_key: "69b239b54afdfe02fecbd10922d53a67"
server_key_iv: "d5a647bcda7a3d0f16cc015c0a5ce0d5"
server_key_version: "encryptionkey@0"
I20260812 16:37:58.554688 32012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "588f801e5007480bbfd2c6860aeb81a2"
format_stamp: "Formatted at 2026-08-12 16:37:58 on dist-test-slave-f7th"
server_key: "69b239b54afdfe02fecbd10922d53a67"
server_key_iv: "d5a647bcda7a3d0f16cc015c0a5ce0d5"
server_key_version: "encryptionkey@0"
I20260812 16:37:58.564468 32012 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20260812 16:37:58.571189 32028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:37:58.573483 32012 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20260812 16:37:58.573823 32012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "588f801e5007480bbfd2c6860aeb81a2"
format_stamp: "Formatted at 2026-08-12 16:37:58 on dist-test-slave-f7th"
server_key: "69b239b54afdfe02fecbd10922d53a67"
server_key_iv: "d5a647bcda7a3d0f16cc015c0a5ce0d5"
server_key_version: "encryptionkey@0"
I20260812 16:37:58.574046 32012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:37:58.599889 32012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:37:58.606215 32012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:37:58.606575 32012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:37:58.643374 32080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:43495 every 8 connection(s)
I20260812 16:37:58.643429 32012 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:43495
I20260812 16:37:58.646013 32012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:37:58.650911 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 32012
I20260812 16:37:58.651355 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:37:58.654662 31990 external_mini_cluster.cc:1428] Setting key 4398139f60d7d428d4e1fb2308ff104d
I20260812 16:37:58.655212 32081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:37:58.674419 32081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2: Bootstrap starting.
I20260812 16:37:58.682494 32081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2: Neither blocks nor log segments found. Creating new log.
I20260812 16:37:58.684664 32081 log.cc:824] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2: Log is configured to *not* fsync() on all Append() calls
I20260812 16:37:58.689765 32081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2: No bootstrap required, opened a new log
Aug 12 16:37:58 dist-test-slave-f7th krb5kdc[31998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:37:58.702116 32081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588f801e5007480bbfd2c6860aeb81a2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43495 } }
I20260812 16:37:58.702935 32081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:37:58.703248 32081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588f801e5007480bbfd2c6860aeb81a2, State: Initialized, Role: FOLLOWER
I20260812 16:37:58.704728 32081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588f801e5007480bbfd2c6860aeb81a2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43495 } }
I20260812 16:37:58.705106 32081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:37:58.705470 32081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:37:58.705829 32081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:37:58.710202 32081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588f801e5007480bbfd2c6860aeb81a2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43495 } }
I20260812 16:37:58.710762 32083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:37:58.662503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.31.61.190:43495)
0812 16:37:58.664058 (+ 1555us) server_negotiation.cc:206] Beginning negotiation
0812 16:37:58.664077 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 16:37:58.667557 (+ 3480us) server_negotiation.cc:407] Connection header received
0812 16:37:58.680690 (+ 13133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:37:58.680760 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:37:58.681521 (+ 761us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:37:58.682152 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:37:58.685814 (+ 3662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:37:58.687770 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:37:58.689956 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:37:58.690673 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:37:58.690955 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:37:58.697875 (+ 6920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:37:58.697953 (+ 78us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:37:58.698001 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:37:58.698055 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:37:58.703217 (+ 5162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:37:58.704560 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:37:58.704569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:37:58.704577 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:37:58.704892 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:37:58.705738 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:37:58.705744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:37:58.705748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:37:58.706728 (+ 980us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:37:58.707137 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0812 16:37:58.708106 (+ 969us) server_negotiation.cc:299] Negotiation successful
0812 16:37:58.708584 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1123,"thread_start_us":752,"threads_started":1}
I20260812 16:37:58.711585 32081 leader_election.cc:302] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 588f801e5007480bbfd2c6860aeb81a2; no voters:
I20260812 16:37:58.712579 32081 leader_election.cc:288] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:37:58.712872 32086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:37:58.714550 32086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [term 1 LEADER]: Becoming Leader. State: Replica: 588f801e5007480bbfd2c6860aeb81a2, State: Running, Role: LEADER
I20260812 16:37:58.715488 32086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588f801e5007480bbfd2c6860aeb81a2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43495 } }
I20260812 16:37:58.716353 32081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:37:58.723217 32088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588f801e5007480bbfd2c6860aeb81a2. Latest consensus state: current_term: 1 leader_uuid: "588f801e5007480bbfd2c6860aeb81a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588f801e5007480bbfd2c6860aeb81a2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43495 } } }
I20260812 16:37:58.724210 32088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [sys.catalog]: This master's current role is: LEADER
I20260812 16:37:58.725389 32087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "588f801e5007480bbfd2c6860aeb81a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588f801e5007480bbfd2c6860aeb81a2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43495 } } }
I20260812 16:37:58.725791 32087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2 [sys.catalog]: This master's current role is: LEADER
I20260812 16:37:58.728996 32095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:37:58.739756 32095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:37:58.757638 32095 catalog_manager.cc:1347] Generated new cluster ID: 5eb5aae3695944f3b158f02e5d8cd09a
I20260812 16:37:58.758028 32095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:37:58.774901 32095 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:37:58.776827 32095 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:37:58.785727 32095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 588f801e5007480bbfd2c6860aeb81a2: Generated new TSK 0
I20260812 16:37:58.786857 32095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:37:58.853170 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43495
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:37:59.382086 32108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:37:59.382578 32108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:37:59.382874 32108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:37:59.396454 32108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:37:59.396608 32108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:37:59.396800 32108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:37:59.412433 32108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:37:59.414111 32108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:37:59.415854 32108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:37:59.432763 32113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:37:59.433163 32116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:37:59.433418 32114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:37:59.434386 32108 server_base.cc:1034] running on GCE node
I20260812 16:37:59.435081 32108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:37:59.436271 32108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:37:59.437558 32108 hybrid_clock.cc:648] HybridClock initialized: now 1786552679437443 us; error 32 us; skew 500 ppm
I20260812 16:37:59.438073 32108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:37:59 dist-test-slave-f7th krb5kdc[31998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:37:59.441958 32108 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:37:59.443681 32108 webserver.cc:460] Webserver started at http://127.31.61.129:39179/ using document root <none> and password file <none>
I20260812 16:37:59.444561 32108 fs_manager.cc:359] Metadata directory not provided
I20260812 16:37:59.444746 32108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:37:59.445081 32108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:37:59.448645 32108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "fd0bdbf80d9d48ba9c932d770d6c7e56"
format_stamp: "Formatted at 2026-08-12 16:37:59 on dist-test-slave-f7th"
server_key: "15cb2b4a6739819201a642500a3369bc"
server_key_iv: "2647f11d5144f31e67d4927d4db4dd17"
server_key_version: "encryptionkey@0"
I20260812 16:37:59.449425 32108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "fd0bdbf80d9d48ba9c932d770d6c7e56"
format_stamp: "Formatted at 2026-08-12 16:37:59 on dist-test-slave-f7th"
server_key: "15cb2b4a6739819201a642500a3369bc"
server_key_iv: "2647f11d5144f31e67d4927d4db4dd17"
server_key_version: "encryptionkey@0"
I20260812 16:37:59.455034 32108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 16:37:59.459757 32124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:37:59.461846 32108 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 16:37:59.462042 32108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "fd0bdbf80d9d48ba9c932d770d6c7e56"
format_stamp: "Formatted at 2026-08-12 16:37:59 on dist-test-slave-f7th"
server_key: "15cb2b4a6739819201a642500a3369bc"
server_key_iv: "2647f11d5144f31e67d4927d4db4dd17"
server_key_version: "encryptionkey@0"
I20260812 16:37:59.462387 32108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:37:59.494776 32108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:37:59.500424 32108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:37:59.500722 32108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:37:59.502250 32108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:37:59.504938 32108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:37:59.505077 32108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:37:59.505282 32108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:37:59.505360 32108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:37:59.562490 32108 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:34273
I20260812 16:37:59.562527 32237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:34273 every 8 connection(s)
I20260812 16:37:59.564100 32108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:37:59.571069 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 32108
I20260812 16:37:59.571656 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:37:59.572646 31990 external_mini_cluster.cc:1428] Setting key 3fe101604d13abb82b8c687a20194396
Aug 12 16:37:59 dist-test-slave-f7th krb5kdc[31998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
I20260812 16:37:59.593206 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:37:59.567756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:35859 (local address 127.31.61.190:43495)
0812 16:37:59.568279 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 16:37:59.568283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:37:59.569775 (+ 1492us) server_negotiation.cc:407] Connection header received
0812 16:37:59.573091 (+ 3316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:37:59.573096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:37:59.573182 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:37:59.573284 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:37:59.575314 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:37:59.576302 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:37:59.578791 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:37:59.579480 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:37:59.579631 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:37:59.584622 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:37:59.584630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:37:59.584634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:37:59.584676 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:37:59.588796 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:37:59.589939 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:37:59.589946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:37:59.589950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:37:59.590115 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:37:59.590762 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:37:59.590767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:37:59.590770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:37:59.591297 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:37:59.591475 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 16:37:59.592469 (+ 994us) server_negotiation.cc:299] Negotiation successful
0812 16:37:59.592778 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":243,"threads_started":1}
I20260812 16:37:59.594086 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:37:59.568374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43495 (local address 127.31.61.129:35859)
0812 16:37:59.569579 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0812 16:37:59.569635 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:37:59.572623 (+ 2988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:37:59.573587 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:37:59.573597 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:37:59.574193 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:37:59.575127 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:37:59.575155 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:37:59.576523 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:37:59.576530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:37:59.578602 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:37:59.578620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:37:59.579764 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:37:59.579770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:37:59.580065 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:37:59.581088 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:37:59.581128 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:37:59.584271 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:37:59.589059 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:37:59.589069 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:37:59.589086 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:37:59.589792 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:37:59.590247 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:37:59.590252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:37:59.590254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:37:59.590633 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:37:59.591541 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:37:59.591549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:37:59.591964 (+ 415us) client_negotiation.cc:769] Sending connection context
0812 16:37:59.592292 (+ 328us) client_negotiation.cc:240] Negotiation successful
0812 16:37:59.592778 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":574,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
I20260812 16:37:59.596328 32238 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43495
I20260812 16:37:59.596781 32238 heartbeater.cc:464] Registering TS with master...
I20260812 16:37:59.597707 32238 heartbeater.cc:511] Master 127.31.61.190:43495 requested a full tablet report, sending...
I20260812 16:37:59.602375 32046 ts_manager.cc:194] Registered new tserver with Master: fd0bdbf80d9d48ba9c932d770d6c7e56 (127.31.61.129:34273)
I20260812 16:37:59.605139 32046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:35859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:37:59.623850 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43495
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:00.113231 32244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:00.113569 32244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:00.113715 32244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:00.122373 32244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:00.122646 32244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:00.122895 32244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:00.135478 32244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:00.136963 32244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:00.138453 32244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:00.155830 32249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:00.162206 32250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:00.162806 32252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:00.164063 32244 server_base.cc:1034] running on GCE node
I20260812 16:38:00.164848 32244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:00.165867 32244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:00.167096 32244 hybrid_clock.cc:648] HybridClock initialized: now 1786552680167042 us; error 44 us; skew 500 ppm
I20260812 16:38:00.167521 32244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:00 dist-test-slave-f7th krb5kdc[31998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:00.171711 32244 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:38:00.173663 32244 webserver.cc:460] Webserver started at http://127.31.61.130:36865/ using document root <none> and password file <none>
I20260812 16:38:00.174580 32244 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:00.174713 32244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:00.175017 32244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:00.178546 32244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "b0e289ec6eec45b6a326a034b51997b4"
format_stamp: "Formatted at 2026-08-12 16:38:00 on dist-test-slave-f7th"
server_key: "7218abf26b54f24796845bea1f16440b"
server_key_iv: "e52c8b8c430ebb280bced008be6d46f9"
server_key_version: "encryptionkey@0"
I20260812 16:38:00.179407 32244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "b0e289ec6eec45b6a326a034b51997b4"
format_stamp: "Formatted at 2026-08-12 16:38:00 on dist-test-slave-f7th"
server_key: "7218abf26b54f24796845bea1f16440b"
server_key_iv: "e52c8b8c430ebb280bced008be6d46f9"
server_key_version: "encryptionkey@0"
I20260812 16:38:00.186416 32244 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 16:38:00.190585 32260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:00.192211 32244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:00.192443 32244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "b0e289ec6eec45b6a326a034b51997b4"
format_stamp: "Formatted at 2026-08-12 16:38:00 on dist-test-slave-f7th"
server_key: "7218abf26b54f24796845bea1f16440b"
server_key_iv: "e52c8b8c430ebb280bced008be6d46f9"
server_key_version: "encryptionkey@0"
I20260812 16:38:00.192706 32244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:00.217411 32244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:00.223225 32244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:00.223770 32244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:00.225168 32244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:00.227182 32244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:00.227279 32244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:00.227519 32244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:00.227626 32244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:00.285377 32244 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:33289
I20260812 16:38:00.285486 32372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:33289 every 8 connection(s)
I20260812 16:38:00.286989 32244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:00.297061 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 32244
I20260812 16:38:00.297263 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:00.297770 31990 external_mini_cluster.cc:1428] Setting key 583281d8417ed86dbcae71c0353c6e21
Aug 12 16:38:00 dist-test-slave-f7th krb5kdc[31998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:00.316788 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:00.290814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43495 (local address 127.31.61.130:33141)
0812 16:38:00.291819 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:00.291914 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:00.294433 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:00.295269 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:00.295278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:00.295944 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:00.296847 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:00.296879 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:00.298187 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:00.298192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:00.300291 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:00.300309 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:00.301176 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:00.301184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:00.301510 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:00.302404 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:00.302568 (+ 164us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:00.305913 (+ 3345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:00.310833 (+ 4920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:00.310844 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:00.310865 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:00.311771 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:00.312445 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:00.312450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:00.312453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:00.312801 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:00.313741 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:00.313751 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:00.314401 (+ 650us) client_negotiation.cc:769] Sending connection context
0812 16:38:00.314722 (+ 321us) client_negotiation.cc:240] Negotiation successful
0812 16:38:00.315273 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":377,"threads_started":1}
I20260812 16:38:00.317776 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:00.290449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:33141 (local address 127.31.61.190:43495)
0812 16:38:00.291176 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:00.291181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:00.292056 (+ 875us) server_negotiation.cc:407] Connection header received
0812 16:38:00.294755 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:00.294761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:00.294838 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:00.295025 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:00.297092 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:00.298020 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:00.300472 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:00.300907 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:00.301033 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:00.306296 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:00.306302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:00.306307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:00.306352 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:00.310607 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:00.311979 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:00.311985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:00.311988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:00.312288 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:00.313055 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:00.313063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:00.313067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:00.313506 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:00.313733 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:00.316777 (+ 3044us) server_negotiation.cc:299] Negotiation successful
0812 16:38:00.317303 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":507,"threads_started":1}
I20260812 16:38:00.319031 32373 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43495
I20260812 16:38:00.319495 32373 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:00.320636 32373 heartbeater.cc:511] Master 127.31.61.190:43495 requested a full tablet report, sending...
I20260812 16:38:00.322582 32046 ts_manager.cc:194] Registered new tserver with Master: b0e289ec6eec45b6a326a034b51997b4 (127.31.61.130:33289)
I20260812 16:38:00.323763 32046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:33141
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:38:00.352741 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43495
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:38:00.609154 32238 heartbeater.cc:503] Master 127.31.61.190:43495 was elected leader, sending a full tablet report...
W20260812 16:38:00.823203 32379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:00.823647 32379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:00.823874 32379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:00.833855 32379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:00.834060 32379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:00.834224 32379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:00.846382 32379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:00.847761 32379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:00.849216 32379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:00.864612 32384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:00.865063 32385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:00.865059 32387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:00.867148 32379 server_base.cc:1034] running on GCE node
I20260812 16:38:00.867939 32379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:00.869112 32379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:00.870360 32379 hybrid_clock.cc:648] HybridClock initialized: now 1786552680870286 us; error 62 us; skew 500 ppm
I20260812 16:38:00.870807 32379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:00 dist-test-slave-f7th krb5kdc[31998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:00.874540 32379 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:38:00.876555 32379 webserver.cc:460] Webserver started at http://127.31.61.131:41843/ using document root <none> and password file <none>
I20260812 16:38:00.877373 32379 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:00.877496 32379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:00.877811 32379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:00.881922 32379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "5b1bc377ef2e43f396e9a18e98456f0b"
format_stamp: "Formatted at 2026-08-12 16:38:00 on dist-test-slave-f7th"
server_key: "4c5bd6007725fe9fd442e95e0337b462"
server_key_iv: "45a63dfbe34b5bdd08d696b87eb8d2f5"
server_key_version: "encryptionkey@0"
I20260812 16:38:00.882761 32379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "5b1bc377ef2e43f396e9a18e98456f0b"
format_stamp: "Formatted at 2026-08-12 16:38:00 on dist-test-slave-f7th"
server_key: "4c5bd6007725fe9fd442e95e0337b462"
server_key_iv: "45a63dfbe34b5bdd08d696b87eb8d2f5"
server_key_version: "encryptionkey@0"
I20260812 16:38:00.889675 32379 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260812 16:38:00.894667 32395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:00.896385 32379 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:38:00.896582 32379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "5b1bc377ef2e43f396e9a18e98456f0b"
format_stamp: "Formatted at 2026-08-12 16:38:00 on dist-test-slave-f7th"
server_key: "4c5bd6007725fe9fd442e95e0337b462"
server_key_iv: "45a63dfbe34b5bdd08d696b87eb8d2f5"
server_key_version: "encryptionkey@0"
I20260812 16:38:00.896811 32379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:00.915706 32379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:00.921029 32379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:00.921424 32379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:00.922966 32379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:00.925490 32379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:00.925596 32379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:00.925724 32379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:00.925797 32379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:00.985986 32379 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:43821
I20260812 16:38:00.986071 32509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:43821 every 8 connection(s)
I20260812 16:38:00.987453 32379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:00.995216 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 32379
I20260812 16:38:00.995466 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:00.996091 31990 external_mini_cluster.cc:1428] Setting key 6671fc2a5d0fd4b5fe68c374291d9e48
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[31998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:01.013523 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:00.991277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:45441 (local address 127.31.61.190:43495)
0812 16:38:00.991938 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:00.991943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:00.993398 (+ 1455us) server_negotiation.cc:407] Connection header received
0812 16:38:00.996073 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:00.996079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:00.996164 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:00.996289 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:00.998233 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:00.999424 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.001261 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.001780 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.001896 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:01.006457 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:01.006463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:01.006466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:01.006502 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:01.009577 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.010554 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.010559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.010561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.010700 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.011294 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.011298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.011301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.011749 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:01.011901 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:01.012917 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0812 16:38:01.013155 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":279,"threads_started":1}
I20260812 16:38:01.014456 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:00.992005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43495 (local address 127.31.61.131:45441)
0812 16:38:00.993135 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:00.993188 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:00.995566 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:00.996609 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:00.996619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:00.997339 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:00.998057 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:00.998086 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:00.999586 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:00.999590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:01.001121 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:01.001135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.002017 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.002026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:01.002372 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:01.003153 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:01.003178 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:01.006158 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:01.009750 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:01.009759 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:01.009774 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:01.010399 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:01.010801 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:01.010804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:01.010806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:01.011153 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:01.011982 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:01.011988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:01.012453 (+ 465us) client_negotiation.cc:769] Sending connection context
0812 16:38:01.012805 (+ 352us) client_negotiation.cc:240] Negotiation successful
0812 16:38:01.013283 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":438,"threads_started":1}
I20260812 16:38:01.016253 32510 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43495
I20260812 16:38:01.016609 32510 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:01.017473 32510 heartbeater.cc:511] Master 127.31.61.190:43495 requested a full tablet report, sending...
I20260812 16:38:01.019119 32046 ts_manager.cc:194] Registered new tserver with Master: 5b1bc377ef2e43f396e9a18e98456f0b (127.31.61.131:43821)
I20260812 16:38:01.020012 32046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:45441
I20260812 16:38:01.026584 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[31998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552681, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[31998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552681, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:01.077062 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:01.058063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.31.61.129:34273)
0812 16:38:01.058702 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:01.058709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:01.058734 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:38:01.058793 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:01.058799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:01.059013 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:01.059147 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:01.061050 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.062192 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.063942 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.064708 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.064826 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:01.070139 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:01.070153 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:01.070169 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:01.070216 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:01.073844 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.074684 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.074692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.074694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.074868 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.075290 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.075294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.075296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.075730 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:01.076059 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:01.076369 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 16:38:01.076696 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":334,"threads_started":1}
W20260812 16:38:01.078583 32152 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56610
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[31998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[31998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:01.128465 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:01.112158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.31.61.129:34273)
0812 16:38:01.112363 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:01.112367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:01.112477 (+ 110us) server_negotiation.cc:407] Connection header received
0812 16:38:01.112693 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:01.112698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:01.112803 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:01.112899 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:01.114734 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.115860 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.117283 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.117758 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.117897 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:01.122674 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:01.122680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:01.122683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:01.122726 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:01.125674 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.126488 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.126493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.126495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.126647 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.127050 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.127054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.127056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.127325 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:01.127527 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:01.127955 (+ 428us) server_negotiation.cc:299] Negotiation successful
0812 16:38:01.128154 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:38:01.131395 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 32108
I20260812 16:38:01.150731 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:01.132634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.31.61.129:34273)
0812 16:38:01.132879 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:01.132885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:01.133147 (+ 262us) server_negotiation.cc:407] Connection header received
0812 16:38:01.133275 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:01.133279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:01.133342 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:01.133421 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:01.134979 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.136036 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.137526 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.138070 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.138175 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:01.142332 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:01.142338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:01.142341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:01.142383 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:01.146876 (+ 4493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.148127 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.148133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.148135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.148375 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.148917 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.148921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.148923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.149288 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:01.149801 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:01.150211 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 16:38:01.150397 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:38:01.151786 32233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:01.241772 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 32108
I20260812 16:38:01.256870 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 32244
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[31998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:38:01.276381 32525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:01.258003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.31.61.130:33289)
0812 16:38:01.258710 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:01.258717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:01.258743 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:38:01.258803 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:01.258810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:01.259022 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:01.259214 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:01.261149 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.262249 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.264191 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.264911 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.265051 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:01.269887 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:01.269901 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:01.269918 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:01.269961 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:01.273004 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.273880 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.273893 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.273895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.274047 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.274576 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.274581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.274583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.275066 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:01.275338 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:01.275768 (+ 430us) server_negotiation.cc:299] Negotiation successful
0812 16:38:01.275988 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":362,"threads_started":1}
I20260812 16:38:01.278105 32368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:01.361563 32373 heartbeater.cc:503] Master 127.31.61.190:43495 was elected leader, sending a full tablet report...
I20260812 16:38:01.362571 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 32244
W20260812 16:38:01.377488 32032 connection.cc:535] server connection from 127.31.61.130:33141 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 16:38:01.377794 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 32379
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[31998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:38:01.396054 32527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:01.379144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.31.61.131:43821)
0812 16:38:01.379870 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:01.379877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:01.379909 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:38:01.379969 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:01.379975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:01.380196 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:01.380329 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:01.381832 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.382905 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.384480 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:01.385107 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:01.385212 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:01.389939 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:01.389956 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:01.389975 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:01.390022 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:01.392906 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.393808 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.393816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.393819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.393979 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:01.394391 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:01.394395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:01.394397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:01.394842 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:01.395112 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:01.395452 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 16:38:01.395758 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":406,"threads_started":1}
I20260812 16:38:01.397346 32505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:01.481055 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 32379
I20260812 16:38:01.495226 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 32012
I20260812 16:38:01.496447 32076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:01.600772 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 32012
2026-08-12T16:38:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4853 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[32532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[32532](info): set up 2 sockets
Aug 12 16:38:01 dist-test-slave-f7th krb5kdc[32532](info): commencing operation
krb5kdc: starting...
W20260812 16:38:02.619572 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:02 dist-test-slave-f7th krb5kdc[32532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:38:02.817355 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:32803
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:32803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:03.143332 32546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:03.143687 32546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:03.143823 32546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:03.151139 32546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:03.151275 32546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:03.151353 32546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:03.151430 32546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:03.151495 32546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:03.161665 32546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:32803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:32803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:03.163118 32546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:03.164496 32546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:03.176407 32551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:03.176407 32554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:03.176661 32552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:03.177940 32546 server_base.cc:1034] running on GCE node
I20260812 16:38:03.178761 32546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:03.180305 32546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:03.181535 32546 hybrid_clock.cc:648] HybridClock initialized: now 1786552683181483 us; error 34 us; skew 500 ppm
I20260812 16:38:03.181869 32546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:03 dist-test-slave-f7th krb5kdc[32532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:03.189141 32546 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:38:03.190788 32546 webserver.cc:460] Webserver started at http://127.31.61.190:35461/ using document root <none> and password file <none>
I20260812 16:38:03.191483 32546 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:03.191591 32546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:03.191951 32546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:03.194973 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "bf4a665433934436a09f52b84a2599f9"
format_stamp: "Formatted at 2026-08-12 16:38:03 on dist-test-slave-f7th"
server_key: "c07ec043f552491e00c5c12c1e2b17bc"
server_key_iv: "95c9aa22031bef574e1925676f2f8692"
server_key_version: "encryptionkey@0"
I20260812 16:38:03.195786 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "bf4a665433934436a09f52b84a2599f9"
format_stamp: "Formatted at 2026-08-12 16:38:03 on dist-test-slave-f7th"
server_key: "c07ec043f552491e00c5c12c1e2b17bc"
server_key_iv: "95c9aa22031bef574e1925676f2f8692"
server_key_version: "encryptionkey@0"
I20260812 16:38:03.201061 32546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 16:38:03.204620 32561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:03.205976 32546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:38:03.206163 32546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "bf4a665433934436a09f52b84a2599f9"
format_stamp: "Formatted at 2026-08-12 16:38:03 on dist-test-slave-f7th"
server_key: "c07ec043f552491e00c5c12c1e2b17bc"
server_key_iv: "95c9aa22031bef574e1925676f2f8692"
server_key_version: "encryptionkey@0"
I20260812 16:38:03.206368 32546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:03.223749 32546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:03.228132 32546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:03.228425 32546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:03.246811 32546 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:32803
I20260812 16:38:03.246915 32612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:32803 every 8 connection(s)
I20260812 16:38:03.248279 32546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:03.252030 32613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:03.255520 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 32546
I20260812 16:38:03.255760 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:03.256398 31990 external_mini_cluster.cc:1428] Setting key ea54ea69df7863342aefeb0634013d96
I20260812 16:38:03.264353 32613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9: Bootstrap starting.
I20260812 16:38:03.268074 32613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:03.269279 32613 log.cc:824] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:03.272662 32613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9: No bootstrap required, opened a new log
Aug 12 16:38:03 dist-test-slave-f7th krb5kdc[32532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:03.281745 32613 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4a665433934436a09f52b84a2599f9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32803 } }
I20260812 16:38:03.282205 32613 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:03.282269 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:03.258475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.31.61.190:32803)
0812 16:38:03.259316 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:03.259332 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:03.259409 (+ 77us) server_negotiation.cc:407] Connection header received
0812 16:38:03.262484 (+ 3075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:03.262517 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:03.263083 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:03.263544 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:03.265216 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:03.267214 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:03.268688 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:03.269725 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:03.269926 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:03.274318 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:03.274326 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:03.274342 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:03.274384 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:03.277589 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:03.278453 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:03.278464 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:03.278471 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:03.278670 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:03.279073 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:03.279077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:03.279079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:03.279757 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:03.279996 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:03.280536 (+ 540us) server_negotiation.cc:299] Negotiation successful
0812 16:38:03.280945 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":325,"threads_started":1}
I20260812 16:38:03.282326 32613 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4a665433934436a09f52b84a2599f9, State: Initialized, Role: FOLLOWER
I20260812 16:38:03.283135 32613 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4a665433934436a09f52b84a2599f9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32803 } }
I20260812 16:38:03.283479 32613 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:03.283695 32613 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:03.283912 32613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:03.286643 32613 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4a665433934436a09f52b84a2599f9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32803 } }
I20260812 16:38:03.287173 32613 leader_election.cc:302] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf4a665433934436a09f52b84a2599f9; no voters:
I20260812 16:38:03.287859 32613 leader_election.cc:288] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:03.288051 32618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:03.289352 32618 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [term 1 LEADER]: Becoming Leader. State: Replica: bf4a665433934436a09f52b84a2599f9, State: Running, Role: LEADER
I20260812 16:38:03.289904 32618 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4a665433934436a09f52b84a2599f9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32803 } }
I20260812 16:38:03.290215 32613 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:03.292862 32619 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf4a665433934436a09f52b84a2599f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4a665433934436a09f52b84a2599f9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32803 } } }
I20260812 16:38:03.292853 32620 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4a665433934436a09f52b84a2599f9. Latest consensus state: current_term: 1 leader_uuid: "bf4a665433934436a09f52b84a2599f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4a665433934436a09f52b84a2599f9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32803 } } }
I20260812 16:38:03.293632 32619 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:03.293678 32620 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:03.296278 32628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:03.300908 32628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:03.310055 32628 catalog_manager.cc:1347] Generated new cluster ID: 1908ae3fa9c4481e8c0e1ea353801953
I20260812 16:38:03.310201 32628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:03.321220 32628 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:03.323148 32628 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:03.336956 32628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf4a665433934436a09f52b84a2599f9: Generated new TSK 0
I20260812 16:38:03.338117 32628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:38:03.392006 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32803
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:03.734589 32640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:03.734918 32640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:03.735064 32640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:03.742908 32640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:03.743065 32640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:03.743234 32640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:03.753969 32640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:03.755389 32640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:03.756914 32640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:03.771663 32645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:03.771701 32646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:03.771723 32648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:03.772326 32640 server_base.cc:1034] running on GCE node
I20260812 16:38:03.773017 32640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:03.774029 32640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:03.775259 32640 hybrid_clock.cc:648] HybridClock initialized: now 1786552683775189 us; error 63 us; skew 500 ppm
I20260812 16:38:03.775660 32640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:03 dist-test-slave-f7th krb5kdc[32532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:03.778908 32640 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:38:03.780545 32640 webserver.cc:460] Webserver started at http://127.31.61.129:38239/ using document root <none> and password file <none>
I20260812 16:38:03.781283 32640 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:03.781410 32640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:03.781728 32640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:03.784778 32640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "53bff130c34e4a6699d14e2146f522bc"
format_stamp: "Formatted at 2026-08-12 16:38:03 on dist-test-slave-f7th"
server_key: "8a3d229c45d0d9461b8d7ee2faeff326"
server_key_iv: "dfe052834caa22b0e2bbc0694abcc9c2"
server_key_version: "encryptionkey@0"
I20260812 16:38:03.785583 32640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "53bff130c34e4a6699d14e2146f522bc"
format_stamp: "Formatted at 2026-08-12 16:38:03 on dist-test-slave-f7th"
server_key: "8a3d229c45d0d9461b8d7ee2faeff326"
server_key_iv: "dfe052834caa22b0e2bbc0694abcc9c2"
server_key_version: "encryptionkey@0"
I20260812 16:38:03.791064 32640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 16:38:03.794705 32655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:03.796188 32640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:38:03.796353 32640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "53bff130c34e4a6699d14e2146f522bc"
format_stamp: "Formatted at 2026-08-12 16:38:03 on dist-test-slave-f7th"
server_key: "8a3d229c45d0d9461b8d7ee2faeff326"
server_key_iv: "dfe052834caa22b0e2bbc0694abcc9c2"
server_key_version: "encryptionkey@0"
I20260812 16:38:03.796572 32640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:03.824720 32640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:03.829501 32640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:03.829773 32640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:03.830880 32640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:03.832633 32640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:03.832728 32640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:03.832919 32640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:03.832996 32640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:03.872586 32640 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:39501
I20260812 16:38:03.872606 300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:39501 every 8 connection(s)
I20260812 16:38:03.874011 32640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:03.883478 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 32640
I20260812 16:38:03.883751 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:03.884536 31990 external_mini_cluster.cc:1428] Setting key a01708b66ffaf36c31a754c8d0c5d90c
Aug 12 16:38:03 dist-test-slave-f7th krb5kdc[32532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:03.899977 303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:03.876851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:50401 (local address 127.31.61.190:32803)
0812 16:38:03.877564 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:03.877573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:03.879558 (+ 1985us) server_negotiation.cc:407] Connection header received
0812 16:38:03.881808 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:03.881813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:03.881932 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:03.882048 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:03.884983 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:03.885911 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:03.887908 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:03.888403 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:03.888545 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:03.893072 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:03.893077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:03.893081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:03.893120 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:03.896218 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:03.897368 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:03.897373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:03.897376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:03.897543 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:03.898040 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:03.898043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:03.898046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:03.898346 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:03.898490 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:03.899312 (+ 822us) server_negotiation.cc:299] Negotiation successful
0812 16:38:03.899582 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":322,"threads_started":1}
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
I20260812 16:38:03.901082 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:03.877954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:32803 (local address 127.31.61.129:50401)
0812 16:38:03.879276 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:03.879343 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:03.881393 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:03.882515 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:03.882533 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:03.883702 (+ 1169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:03.884674 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:03.884713 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:03.886109 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:03.886114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:03.887720 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:03.887738 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:03.888685 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:03.888695 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:03.888974 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:03.889806 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:03.889834 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:03.892731 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:03.896419 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:03.896431 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:03.896479 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:03.897197 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:03.897683 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:03.897689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:03.897693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:03.897908 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:03.898490 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:03.898501 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:03.898882 (+ 381us) client_negotiation.cc:769] Sending connection context
0812 16:38:03.899159 (+ 277us) client_negotiation.cc:240] Negotiation successful
0812 16:38:03.899772 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":522,"threads_started":1}
I20260812 16:38:03.902899 301 heartbeater.cc:347] Connected to a master server at 127.31.61.190:32803
I20260812 16:38:03.903255 301 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:03.904122 301 heartbeater.cc:511] Master 127.31.61.190:32803 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
I20260812 16:38:03.906752 32578 ts_manager.cc:194] Registered new tserver with Master: 53bff130c34e4a6699d14e2146f522bc (127.31.61.129:39501)
I20260812 16:38:03.908917 32578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:50401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:38:03.927438 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32803
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:04.262146 307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:04.262458 307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:04.262593 307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:04.270187 307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:04.270321 307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:04.270483 307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:04.280913 307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:04.282323 307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:04.283773 307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:04.298357 315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:04.298350 313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:04.298393 312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:04.304623 307 server_base.cc:1034] running on GCE node
I20260812 16:38:04.305486 307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:04.306839 307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:04.308072 307 hybrid_clock.cc:648] HybridClock initialized: now 1786552684308017 us; error 57 us; skew 500 ppm
I20260812 16:38:04.308413 307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:04 dist-test-slave-f7th krb5kdc[32532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552684, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:04.312953 307 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:38:04.314672 307 webserver.cc:460] Webserver started at http://127.31.61.130:33677/ using document root <none> and password file <none>
I20260812 16:38:04.315449 307 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:04.315593 307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:04.315971 307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:04.319108 307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "5bb2f12e5304464b91dd0e930b3439b3"
format_stamp: "Formatted at 2026-08-12 16:38:04 on dist-test-slave-f7th"
server_key: "cb15ef97ca7b149aed3eb21caef6f50b"
server_key_iv: "eb3d59bf7771b2d54e99be841f6e5afb"
server_key_version: "encryptionkey@0"
I20260812 16:38:04.320046 307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "5bb2f12e5304464b91dd0e930b3439b3"
format_stamp: "Formatted at 2026-08-12 16:38:04 on dist-test-slave-f7th"
server_key: "cb15ef97ca7b149aed3eb21caef6f50b"
server_key_iv: "eb3d59bf7771b2d54e99be841f6e5afb"
server_key_version: "encryptionkey@0"
I20260812 16:38:04.325855 307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 16:38:04.329677 322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:04.331146 307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:38:04.331329 307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "5bb2f12e5304464b91dd0e930b3439b3"
format_stamp: "Formatted at 2026-08-12 16:38:04 on dist-test-slave-f7th"
server_key: "cb15ef97ca7b149aed3eb21caef6f50b"
server_key_iv: "eb3d59bf7771b2d54e99be841f6e5afb"
server_key_version: "encryptionkey@0"
I20260812 16:38:04.331542 307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:04.359793 307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:04.364099 307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:04.364385 307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:04.365357 307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:04.366762 307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:04.366835 307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:04.366947 307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:04.366997 307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:04.400857 307 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:39807
I20260812 16:38:04.400928 436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:39807 every 8 connection(s)
I20260812 16:38:04.402652 307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:04.407892 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 307
I20260812 16:38:04.408159 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:04.408663 31990 external_mini_cluster.cc:1428] Setting key e13fc5bde0513eb0c714983684dcdf21
Aug 12 16:38:04 dist-test-slave-f7th krb5kdc[32532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552684, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:04.425519 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:04.405782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:59983 (local address 127.31.61.190:32803)
0812 16:38:04.406237 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:04.406242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:04.407287 (+ 1045us) server_negotiation.cc:407] Connection header received
0812 16:38:04.409946 (+ 2659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:04.409952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:04.410048 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:04.410149 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:04.412033 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.412808 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.414488 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.414924 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.415042 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:04.418884 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:04.418889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:04.418893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:04.418929 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:04.422011 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:04.423040 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:04.423045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:04.423047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:04.423188 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:04.423659 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:04.423663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:04.423665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:04.423956 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:04.424103 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:04.424862 (+ 759us) server_negotiation.cc:299] Negotiation successful
0812 16:38:04.425126 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
I20260812 16:38:04.426577 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:04.406345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:32803 (local address 127.31.61.130:59983)
0812 16:38:04.407095 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:04.407153 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:04.409516 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:04.410472 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:04.410484 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:04.411137 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:04.411832 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:04.411857 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.412982 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.412986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:04.414313 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:04.414333 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.415167 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.415175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:04.415404 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:04.416138 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:04.416166 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:04.418606 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:04.422203 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:04.422212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:04.422228 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:04.422887 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:04.423303 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:04.423307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:04.423309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:04.423513 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:04.424097 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:04.424106 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:04.424473 (+ 367us) client_negotiation.cc:769] Sending connection context
0812 16:38:04.424726 (+ 253us) client_negotiation.cc:240] Negotiation successful
0812 16:38:04.425327 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":293,"threads_started":1}
I20260812 16:38:04.428323 437 heartbeater.cc:347] Connected to a master server at 127.31.61.190:32803
I20260812 16:38:04.428695 437 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
I20260812 16:38:04.429530 437 heartbeater.cc:511] Master 127.31.61.190:32803 requested a full tablet report, sending...
I20260812 16:38:04.431569 32578 ts_manager.cc:194] Registered new tserver with Master: 5bb2f12e5304464b91dd0e930b3439b3 (127.31.61.130:39807)
I20260812 16:38:04.432680 32578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:59983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:38:04.455338 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32803
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:04.794226 443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:04.794548 443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:04.794687 443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:04.802152 443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:04.802276 443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:04.802426 443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:04.813802 443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:04.815259 443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:04.816768 443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:04.832919 448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:04.833657 449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:04.833915 451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:04.834812 443 server_base.cc:1034] running on GCE node
I20260812 16:38:04.835740 443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:04.836957 443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:04.838187 443 hybrid_clock.cc:648] HybridClock initialized: now 1786552684838133 us; error 43 us; skew 500 ppm
I20260812 16:38:04.838529 443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:04 dist-test-slave-f7th krb5kdc[32532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552684, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:04.842777 443 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:38:04.844689 443 webserver.cc:460] Webserver started at http://127.31.61.131:35375/ using document root <none> and password file <none>
I20260812 16:38:04.845503 443 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:04.845705 443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:04.846087 443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:04.849175 443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "18c74cb618f94a35953a12800cfdf621"
format_stamp: "Formatted at 2026-08-12 16:38:04 on dist-test-slave-f7th"
server_key: "6db01b654daaf9ea92b90d6bc00d9887"
server_key_iv: "5db07cd9eec56d295cd3871bf852205c"
server_key_version: "encryptionkey@0"
I20260812 16:38:04.850113 443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "18c74cb618f94a35953a12800cfdf621"
format_stamp: "Formatted at 2026-08-12 16:38:04 on dist-test-slave-f7th"
server_key: "6db01b654daaf9ea92b90d6bc00d9887"
server_key_iv: "5db07cd9eec56d295cd3871bf852205c"
server_key_version: "encryptionkey@0"
I20260812 16:38:04.856182 443 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 16:38:04.860493 459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:04.862124 443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:04.862354 443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "18c74cb618f94a35953a12800cfdf621"
format_stamp: "Formatted at 2026-08-12 16:38:04 on dist-test-slave-f7th"
server_key: "6db01b654daaf9ea92b90d6bc00d9887"
server_key_iv: "5db07cd9eec56d295cd3871bf852205c"
server_key_version: "encryptionkey@0"
I20260812 16:38:04.862643 443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:04.881407 443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:04.886696 443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:04.886987 443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:04.888043 443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:04.889668 443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:04.889750 443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:04.889917 443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:04.889981 443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:04.912590 301 heartbeater.cc:503] Master 127.31.61.190:32803 was elected leader, sending a full tablet report...
I20260812 16:38:04.940729 443 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:34991
I20260812 16:38:04.940718 573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:34991 every 8 connection(s)
I20260812 16:38:04.942238 443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:04.945472 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 443
I20260812 16:38:04.945716 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:04.946171 31990 external_mini_cluster.cc:1428] Setting key 479a314f6780d3c0b8932741ea27b2ad
Aug 12 16:38:04 dist-test-slave-f7th krb5kdc[32532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552684, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:04.969591 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:04.945362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:52839 (local address 127.31.61.190:32803)
0812 16:38:04.952168 (+ 6806us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:04.952173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:04.952188 (+ 15us) server_negotiation.cc:407] Connection header received
0812 16:38:04.952234 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:04.952237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:04.952301 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:04.952389 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:04.954125 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.955227 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.957256 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.957703 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.957810 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:04.962727 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:04.962733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:04.962737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:04.962773 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:04.965618 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:04.966527 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:04.966531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:04.966533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:04.966723 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:04.967236 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:04.967240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:04.967242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:04.967533 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:04.967782 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:04.968716 (+ 934us) server_negotiation.cc:299] Negotiation successful
0812 16:38:04.968956 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6703,"thread_start_us":6503,"threads_started":1}
I20260812 16:38:04.970589 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:04.945914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:32803 (local address 127.31.61.131:52839)
0812 16:38:04.947540 (+ 1626us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:04.947633 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:04.950332 (+ 2699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:04.952686 (+ 2354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:04.952696 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:04.953257 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:04.953941 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:04.953963 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.955374 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.955378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:04.957109 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:04.957123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.957952 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.957959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:04.958170 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:04.959033 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:04.959064 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:04.962492 (+ 3428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:04.965775 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:04.965784 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:04.965800 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:04.966342 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:04.966831 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:04.966834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:04.966837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:04.967114 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:04.967730 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:04.967739 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:04.968159 (+ 420us) client_negotiation.cc:769] Sending connection context
0812 16:38:04.968561 (+ 402us) client_negotiation.cc:240] Negotiation successful
0812 16:38:04.969209 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1273,"thread_start_us":553,"threads_started":1}
I20260812 16:38:04.972067 574 heartbeater.cc:347] Connected to a master server at 127.31.61.190:32803
I20260812 16:38:04.972402 574 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:04.973163 574 heartbeater.cc:511] Master 127.31.61.190:32803 requested a full tablet report, sending...
I20260812 16:38:04.974980 32578 ts_manager.cc:194] Registered new tserver with Master: 18c74cb618f94a35953a12800cfdf621 (127.31.61.131:34991)
I20260812 16:38:04.976007 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:38:04.976068 32578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:52839
I20260812 16:38:05.001281 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:04.984356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.31.61.190:32803)
0812 16:38:04.984655 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:04.984663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:04.985209 (+ 546us) server_negotiation.cc:407] Connection header received
0812 16:38:04.985391 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:04.985396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:04.985466 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:04.985552 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:04.987409 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.988570 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.990336 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:04.990829 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:04.990949 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:04.994135 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:04.994141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:04.994144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:04.994186 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:04.997851 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:04.998629 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:04.998635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:04.998638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:04.998767 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:04.999313 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:04.999317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:04.999320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:04.999825 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:05.000139 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.000571 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.000866 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 16:38:05.011783 32578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:05.022068 32578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:05.058472 589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.040772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:34991 (local address 127.0.0.1:40880)
0812 16:38:05.043858 (+ 3086us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:05.043891 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:05.044087 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:05.045829 (+ 1742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:05.045835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:05.045873 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:05.046918 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.046963 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.050658 (+ 3695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.050664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.054932 (+ 4268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.054945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.057170 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.057177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.057392 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.057452 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 16:38:05.057597 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 16:38:05.057796 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2847,"mutex_wait_us":233,"thread_start_us":3517,"threads_started":1}
I20260812 16:38:05.059309 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.041034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:39501 (local address 127.0.0.1:49476)
0812 16:38:05.044700 (+ 3666us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:05.044718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:05.044867 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:05.045695 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:05.045704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:05.045739 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:05.047438 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.047455 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.050680 (+ 3225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.050686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.054810 (+ 4124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.054829 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.058528 (+ 3699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.058535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.058692 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.058725 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:38:05.058787 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 16:38:05.058914 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3516,"mutex_wait_us":953,"thread_start_us":2301,"threads_started":1}
I20260812 16:38:05.060389 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.041089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.31.61.131:34991)
0812 16:38:05.044690 (+ 3601us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.044808 (+ 118us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.044845 (+ 37us) server_negotiation.cc:407] Connection header received
0812 16:38:05.044956 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.044965 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.045429 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.046621 (+ 1192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:05.048313 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.050251 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.055120 (+ 4869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.056854 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.059331 (+ 2477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.059791 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.059870 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.060045 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3355,"thread_start_us":2317,"threads_started":1}
I20260812 16:38:05.060395 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.041607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.31.61.129:39501)
0812 16:38:05.043719 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.043728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.044988 (+ 1260us) server_negotiation.cc:407] Connection header received
0812 16:38:05.045134 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.045143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.045400 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.046108 (+ 708us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:05.047741 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.050331 (+ 2590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.055091 (+ 4760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.058260 (+ 3169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.059518 (+ 1258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.059785 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.059879 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.060045 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":759,"threads_started":1}
I20260812 16:38:05.061344 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.044685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:39807 (local address 127.0.0.1:59118)
0812 16:38:05.047935 (+ 3250us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:05.047955 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:05.048110 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:05.050132 (+ 2022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:05.050139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:05.050180 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:05.053227 (+ 3047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.053246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.055553 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.055559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.058130 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.058148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.060585 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.060594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.060772 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.060805 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:38:05.060884 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 16:38:05.061026 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2962,"thread_start_us":2754,"threads_started":1}
I20260812 16:38:05.062129 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.044900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.31.61.130:39807)
0812 16:38:05.048300 (+ 3400us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.048420 (+ 120us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.048503 (+ 83us) server_negotiation.cc:407] Connection header received
0812 16:38:05.048672 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.048682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.049067 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.049782 (+ 715us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:05.053520 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.055309 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.058390 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.060012 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.060149 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.060337 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.061636 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.061830 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3225,"thread_start_us":3099,"threads_started":1}
I20260812 16:38:05.064769 32703 tablet_service.cc:1467] Processing CreateTablet for tablet 367443fc212e482a990c7c90fbff4a13 (DEFAULT_TABLE table=test-table [id=e527fa89bb1c44de9fa14f08e7d54cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:05.065778 507 tablet_service.cc:1467] Processing CreateTablet for tablet 367443fc212e482a990c7c90fbff4a13 (DEFAULT_TABLE table=test-table [id=e527fa89bb1c44de9fa14f08e7d54cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:05.066251 32703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367443fc212e482a990c7c90fbff4a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:05.066958 507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367443fc212e482a990c7c90fbff4a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:05.069196 370 tablet_service.cc:1467] Processing CreateTablet for tablet 367443fc212e482a990c7c90fbff4a13 (DEFAULT_TABLE table=test-table [id=e527fa89bb1c44de9fa14f08e7d54cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:05.070722 370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367443fc212e482a990c7c90fbff4a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:05.082675 594 tablet_bootstrap.cc:492] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3: Bootstrap starting.
I20260812 16:38:05.085168 595 tablet_bootstrap.cc:492] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc: Bootstrap starting.
I20260812 16:38:05.087342 594 tablet_bootstrap.cc:654] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:05.088826 596 tablet_bootstrap.cc:492] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621: Bootstrap starting.
I20260812 16:38:05.089304 594 log.cc:824] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:05.091390 595 tablet_bootstrap.cc:654] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:05.092806 595 log.cc:824] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:05.093199 596 tablet_bootstrap.cc:654] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:05.093923 594 tablet_bootstrap.cc:492] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3: No bootstrap required, opened a new log
I20260812 16:38:05.094321 594 ts_tablet_manager.cc:1397] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260812 16:38:05.094851 596 log.cc:824] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:05.096474 595 tablet_bootstrap.cc:492] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc: No bootstrap required, opened a new log
I20260812 16:38:05.096789 595 ts_tablet_manager.cc:1397] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260812 16:38:05.098317 596 tablet_bootstrap.cc:492] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621: No bootstrap required, opened a new log
I20260812 16:38:05.098661 596 ts_tablet_manager.cc:1397] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 16:38:05.105576 595 raft_consensus.cc:348] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } }
I20260812 16:38:05.105966 595 raft_consensus.cc:374] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:05.106186 595 raft_consensus.cc:729] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53bff130c34e4a6699d14e2146f522bc, State: Initialized, Role: FOLLOWER
I20260812 16:38:05.106873 595 consensus_queue.cc:260] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } }
I20260812 16:38:05.106719 594 raft_consensus.cc:348] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } }
I20260812 16:38:05.107218 594 raft_consensus.cc:374] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:05.107421 594 raft_consensus.cc:729] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb2f12e5304464b91dd0e930b3439b3, State: Initialized, Role: FOLLOWER
I20260812 16:38:05.108165 594 consensus_queue.cc:260] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } }
I20260812 16:38:05.109340 595 ts_tablet_manager.cc:1428] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260812 16:38:05.111129 596 raft_consensus.cc:348] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } }
I20260812 16:38:05.111725 596 raft_consensus.cc:374] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:05.111878 596 raft_consensus.cc:729] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c74cb618f94a35953a12800cfdf621, State: Initialized, Role: FOLLOWER
I20260812 16:38:05.112721 596 consensus_queue.cc:260] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } }
I20260812 16:38:05.114481 574 heartbeater.cc:503] Master 127.31.61.190:32803 was elected leader, sending a full tablet report...
I20260812 16:38:05.115061 596 ts_tablet_manager.cc:1428] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260812 16:38:05.116405 437 heartbeater.cc:503] Master 127.31.61.190:32803 was elected leader, sending a full tablet report...
I20260812 16:38:05.116931 594 ts_tablet_manager.cc:1428] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3: Time spent starting tablet: real 0.022s user 0.017s sys 0.005s
W20260812 16:38:05.126866 302 tablet.cc:2369] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:38:05.155334 438 tablet.cc:2369] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:38:05.194913 576 tablet.cc:2369] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:38:05.325263 601 raft_consensus.cc:482] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:05.326051 601 raft_consensus.cc:504] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } }
I20260812 16:38:05.328761 601 leader_election.cc:288] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c74cb618f94a35953a12800cfdf621 (127.31.61.131:34991), 53bff130c34e4a6699d14e2146f522bc (127.31.61.129:39501)
I20260812 16:38:05.339882 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.329377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:49827 (local address 127.31.61.131:34991)
0812 16:38:05.329646 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.329651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.330640 (+ 989us) server_negotiation.cc:407] Connection header received
0812 16:38:05.330941 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.330945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.331018 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.331102 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:05.332663 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.333865 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.337474 (+ 3609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.338670 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.338833 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.338899 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.339372 (+ 473us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.339527 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:38:05.339953 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.329269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:34991 (local address 127.31.61.130:49827)
0812 16:38:05.330436 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:05.330452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:05.330783 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:05.331497 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:05.331500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:05.331520 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:05.332464 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.332483 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.334060 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.334065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.337183 (+ 3118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.337199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.338931 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.338939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.339142 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.339181 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 16:38:05.339294 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 16:38:05.339445 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":44,"thread_start_us":652,"threads_started":1}
I20260812 16:38:05.340667 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.329227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:59481 (local address 127.31.61.129:39501)
0812 16:38:05.329499 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.329505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.331009 (+ 1504us) server_negotiation.cc:407] Connection header received
0812 16:38:05.331136 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.331141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.331218 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.331311 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:05.332852 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.334105 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.337808 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.339211 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.339428 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.339505 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.340158 (+ 653us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.340323 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:38:05.341358 604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.329459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:39501 (local address 127.31.61.130:59481)
0812 16:38:05.330903 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:05.330918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:05.331032 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:05.331360 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:05.331363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:05.331426 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:05.332599 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.332621 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.334238 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.334243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.337447 (+ 3204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:05.337473 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.339655 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.339663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:05.339949 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.339983 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:38:05.340848 (+ 865us) client_negotiation.cc:240] Negotiation successful
0812 16:38:05.340988 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"mutex_wait_us":31,"thread_start_us":970,"threads_started":1}
I20260812 16:38:05.341385 529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367443fc212e482a990c7c90fbff4a13" candidate_uuid: "5bb2f12e5304464b91dd0e930b3439b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c74cb618f94a35953a12800cfdf621" is_pre_election: true
I20260812 16:38:05.342298 529 raft_consensus.cc:2454] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bb2f12e5304464b91dd0e930b3439b3 in term 0.
I20260812 16:38:05.343187 326 leader_election.cc:302] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c74cb618f94a35953a12800cfdf621, 5bb2f12e5304464b91dd0e930b3439b3; no voters:
I20260812 16:38:05.343392 32724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367443fc212e482a990c7c90fbff4a13" candidate_uuid: "5bb2f12e5304464b91dd0e930b3439b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53bff130c34e4a6699d14e2146f522bc" is_pre_election: true
I20260812 16:38:05.343677 601 raft_consensus.cc:2781] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:05.343794 601 raft_consensus.cc:482] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:05.343889 601 raft_consensus.cc:3037] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:05.344019 32724 raft_consensus.cc:2454] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bb2f12e5304464b91dd0e930b3439b3 in term 0.
I20260812 16:38:05.346773 601 raft_consensus.cc:504] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } }
I20260812 16:38:05.347554 601 leader_election.cc:288] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [CANDIDATE]: Term 1 election: Requested vote from peers 18c74cb618f94a35953a12800cfdf621 (127.31.61.131:34991), 53bff130c34e4a6699d14e2146f522bc (127.31.61.129:39501)
I20260812 16:38:05.348390 529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367443fc212e482a990c7c90fbff4a13" candidate_uuid: "5bb2f12e5304464b91dd0e930b3439b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c74cb618f94a35953a12800cfdf621"
I20260812 16:38:05.348500 32724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367443fc212e482a990c7c90fbff4a13" candidate_uuid: "5bb2f12e5304464b91dd0e930b3439b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53bff130c34e4a6699d14e2146f522bc"
I20260812 16:38:05.348608 529 raft_consensus.cc:3037] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:05.348805 32724 raft_consensus.cc:3037] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:05.351424 529 raft_consensus.cc:2454] T 367443fc212e482a990c7c90fbff4a13 P 18c74cb618f94a35953a12800cfdf621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bb2f12e5304464b91dd0e930b3439b3 in term 1.
I20260812 16:38:05.352073 32724 raft_consensus.cc:2454] T 367443fc212e482a990c7c90fbff4a13 P 53bff130c34e4a6699d14e2146f522bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bb2f12e5304464b91dd0e930b3439b3 in term 1.
I20260812 16:38:05.352197 326 leader_election.cc:302] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c74cb618f94a35953a12800cfdf621, 5bb2f12e5304464b91dd0e930b3439b3; no voters:
I20260812 16:38:05.352725 601 raft_consensus.cc:2781] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:05.353751 601 raft_consensus.cc:686] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [term 1 LEADER]: Becoming Leader. State: Replica: 5bb2f12e5304464b91dd0e930b3439b3, State: Running, Role: LEADER
I20260812 16:38:05.354276 601 consensus_queue.cc:237] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } }
I20260812 16:38:05.361181 32578 catalog_manager.cc:5495] T 367443fc212e482a990c7c90fbff4a13 P 5bb2f12e5304464b91dd0e930b3439b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bb2f12e5304464b91dd0e930b3439b3 (127.31.61.130). New cstate: current_term: 1 leader_uuid: "5bb2f12e5304464b91dd0e930b3439b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c74cb618f94a35953a12800cfdf621" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53bff130c34e4a6699d14e2146f522bc" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bb2f12e5304464b91dd0e930b3439b3" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39807 } health_report { overall_health: HEALTHY } } }
Aug 12 16:38:05 dist-test-slave-f7th krb5kdc[32532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:05.429900 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.411561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.31.61.129:39501)
0812 16:38:05.411787 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.411794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.412534 (+ 740us) server_negotiation.cc:407] Connection header received
0812 16:38:05.412749 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.412754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.412824 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.412901 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:05.415076 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.416108 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.417578 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.418245 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.418358 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.423024 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:05.423041 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:05.423057 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:05.423109 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:05.426524 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.427378 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.427386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.427389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.427689 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.428057 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.428060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.428063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.428512 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:05.428733 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.429350 (+ 617us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.429611 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Aug 12 16:38:05 dist-test-slave-f7th krb5kdc[32532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:38:05 dist-test-slave-f7th krb5kdc[32532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:05.482555 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.466077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.31.61.129:39501)
0812 16:38:05.466301 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.466309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.466711 (+ 402us) server_negotiation.cc:407] Connection header received
0812 16:38:05.466859 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.466863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.466930 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.467018 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:05.468942 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.469722 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.471194 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.471838 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.471967 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.476486 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:05.476492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:05.476495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:05.476532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:05.479935 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.480607 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.480611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.480614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.480782 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.481201 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.481205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.481207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.481474 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:05.481748 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.481995 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.482212 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 16:38:05 dist-test-slave-f7th krb5kdc[32532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:38:05 dist-test-slave-f7th krb5kdc[32532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:05.531342 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.514687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.31.61.129:39501)
0812 16:38:05.514913 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.514919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.515116 (+ 197us) server_negotiation.cc:407] Connection header received
0812 16:38:05.515258 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.515262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.515329 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.515424 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:05.517167 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.517880 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.519253 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.519768 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.519946 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.524815 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:05.524821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:05.524824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:05.524875 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:05.527643 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.528461 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.528470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.528472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.528618 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.529414 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.529419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.529421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.529842 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:05.530235 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.530603 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.530875 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260812 16:38:05.533996 32683 tablet_service.cc:2908] Rejecting scan request for tablet 367443fc212e482a990c7c90fbff4a13: Uninitialized: clean time has not yet been initialized
I20260812 16:38:05.538200 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 32640
I20260812 16:38:05.554456 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.539205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.31.61.129:39501)
0812 16:38:05.539435 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.539442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.539703 (+ 261us) server_negotiation.cc:407] Connection header received
0812 16:38:05.539800 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.539804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.539892 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.539998 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:05.541515 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.542456 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.543922 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.544550 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.544681 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.547492 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:05.547498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:05.547501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:05.547537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:05.550421 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.551755 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.551760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.551763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.551910 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.552587 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.552601 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.552604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.552937 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:05.553568 (+ 631us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.553820 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.554114 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 16:38:05.555517 32764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:05.648797 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 32640
I20260812 16:38:05.663445 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 307
Aug 12 16:38:05 dist-test-slave-f7th krb5kdc[32532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:38:05.678104 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.664190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.31.61.130:39807)
0812 16:38:05.664675 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.664679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.664696 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:38:05.664752 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.664756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.664816 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.664910 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:05.666028 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.666754 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.668125 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.668597 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.668712 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.672551 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:05.672566 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:05.672583 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:05.672623 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:05.675265 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.676023 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.676030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.676033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.676182 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.676627 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.676631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.676633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.677044 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:05.677266 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.677642 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.677821 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":268,"threads_started":1}
I20260812 16:38:05.678814 432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:05.770123 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 307
I20260812 16:38:05.785043 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 443
Aug 12 16:38:05 dist-test-slave-f7th krb5kdc[32532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:38:05.800190 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:05.785831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.31.61.131:34991)
0812 16:38:05.786003 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:05.786010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:05.786033 (+ 23us) server_negotiation.cc:407] Connection header received
0812 16:38:05.786244 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:05.786250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:05.786344 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:05.786447 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:05.787711 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.788453 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.789749 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:05.790196 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:05.790308 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:05.794507 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:05.794523 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:05.794540 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:05.794578 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:05.797441 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.798122 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.798129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.798132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.798269 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:05.798624 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:05.798627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:05.798629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:05.798995 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:05.799273 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:05.799591 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 16:38:05.799844 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 16:38:05.800840 569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:05.890626 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 443
I20260812 16:38:05.906020 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 32546
I20260812 16:38:05.907795 32608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:06.020227 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 32546
2026-08-12T16:38:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4418 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:06 dist-test-slave-f7th krb5kdc[625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:06 dist-test-slave-f7th krb5kdc[625](info): set up 2 sockets
Aug 12 16:38:06 dist-test-slave-f7th krb5kdc[625](info): commencing operation
krb5kdc: starting...
W20260812 16:38:07.035259 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:07 dist-test-slave-f7th krb5kdc[625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:38:07.231150 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:45657
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:07.572901 638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:07.573230 638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:07.573354 638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:07.580668 638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:07.580817 638 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 16:38:07.580888 638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:07.580979 638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:07.581046 638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:07.581159 638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:07.591243 638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:45657
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:07.592622 638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:07.594004 638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:07.606400 647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:07.606483 644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:07.606644 643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:07.607416 638 server_base.cc:1034] running on GCE node
I20260812 16:38:07.608268 638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:07.609959 638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:07.611205 638 hybrid_clock.cc:648] HybridClock initialized: now 1786552687611140 us; error 34 us; skew 500 ppm
I20260812 16:38:07.611552 638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:07 dist-test-slave-f7th krb5kdc[625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552687, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:07.614573 638 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:38:07.616117 638 webserver.cc:460] Webserver started at http://127.31.61.190:34831/ using document root <none> and password file <none>
I20260812 16:38:07.616828 638 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:07.616933 638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:07.617244 638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:07.620280 638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "616192fc9c0c43469aaa0e52e4575b6f"
format_stamp: "Formatted at 2026-08-12 16:38:07 on dist-test-slave-f7th"
server_key: "aaf1bfd656206b3beed2915beace2b72"
server_key_iv: "d2ee73320567aac9b608b81b4dd2cf63"
server_key_version: "encryptionkey@0"
I20260812 16:38:07.621078 638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "616192fc9c0c43469aaa0e52e4575b6f"
format_stamp: "Formatted at 2026-08-12 16:38:07 on dist-test-slave-f7th"
server_key: "aaf1bfd656206b3beed2915beace2b72"
server_key_iv: "d2ee73320567aac9b608b81b4dd2cf63"
server_key_version: "encryptionkey@0"
I20260812 16:38:07.626487 638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:38:07.630651 654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:07.632462 638 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 16:38:07.632650 638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "616192fc9c0c43469aaa0e52e4575b6f"
format_stamp: "Formatted at 2026-08-12 16:38:07 on dist-test-slave-f7th"
server_key: "aaf1bfd656206b3beed2915beace2b72"
server_key_iv: "d2ee73320567aac9b608b81b4dd2cf63"
server_key_version: "encryptionkey@0"
I20260812 16:38:07.632890 638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:07.673141 638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:07.677580 638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:07.677851 638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:07.700127 638 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:45657
I20260812 16:38:07.700392 716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:45657 every 8 connection(s)
I20260812 16:38:07.701599 638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:07.705711 717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:07.711416 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 638
I20260812 16:38:07.711663 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:07.712306 31990 external_mini_cluster.cc:1428] Setting key 80db95fc7c0a4111c4f8bb71c0e40158
I20260812 16:38:07.718272 717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f: Bootstrap starting.
I20260812 16:38:07.722223 717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:07.723345 717 log.cc:824] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:07.726348 717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f: No bootstrap required, opened a new log
Aug 12 16:38:07 dist-test-slave-f7th krb5kdc[625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:07.736348 717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616192fc9c0c43469aaa0e52e4575b6f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45657 } }
I20260812 16:38:07.736842 717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:07.736949 717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616192fc9c0c43469aaa0e52e4575b6f, State: Initialized, Role: FOLLOWER
I20260812 16:38:07.737679 717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616192fc9c0c43469aaa0e52e4575b6f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45657 } }
I20260812 16:38:07.737982 717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:07.738148 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:07.714478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.31.61.190:45657)
0812 16:38:07.715371 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:07.715392 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:07.715447 (+ 55us) server_negotiation.cc:407] Connection header received
0812 16:38:07.717909 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:07.717942 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:07.718504 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:07.718986 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:07.720380 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:07.722368 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:07.723801 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:07.724637 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:07.724819 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:07.728796 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:07.728811 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:07.728837 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:07.728884 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:07.733117 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:07.733974 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:07.733984 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:07.733994 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:07.734247 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:07.734614 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:07.734619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:07.734623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:07.735439 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:07.735789 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:07.736186 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 16:38:07.736701 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":374,"threads_started":1}
I20260812 16:38:07.738170 717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:07.738349 717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:07.740887 717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616192fc9c0c43469aaa0e52e4575b6f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45657 } }
I20260812 16:38:07.741480 717 leader_election.cc:302] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 616192fc9c0c43469aaa0e52e4575b6f; no voters:
I20260812 16:38:07.742048 717 leader_election.cc:288] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:07.742182 722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:07.742848 722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [term 1 LEADER]: Becoming Leader. State: Replica: 616192fc9c0c43469aaa0e52e4575b6f, State: Running, Role: LEADER
I20260812 16:38:07.743445 722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616192fc9c0c43469aaa0e52e4575b6f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45657 } }
I20260812 16:38:07.744122 717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:07.746652 724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 616192fc9c0c43469aaa0e52e4575b6f. Latest consensus state: current_term: 1 leader_uuid: "616192fc9c0c43469aaa0e52e4575b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616192fc9c0c43469aaa0e52e4575b6f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45657 } } }
I20260812 16:38:07.747488 724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:07.747947 723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "616192fc9c0c43469aaa0e52e4575b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616192fc9c0c43469aaa0e52e4575b6f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45657 } } }
I20260812 16:38:07.748313 723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:07.749948 731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:07.755390 731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:07.763952 731 catalog_manager.cc:1347] Generated new cluster ID: 3d006d24115c4ba79e43821071c5492c
I20260812 16:38:07.764081 731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:07.776109 731 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:07.777503 731 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:07.784915 731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f: Generated new TSK 0
I20260812 16:38:07.786024 731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:38:07.844936 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45657
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:08.174381 743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:08.174685 743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:08.174818 743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:08.182384 743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:08.182511 743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:08.182618 743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:38:08.182730 743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:08.192698 743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.61.190:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:08.194054 743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:08.195626 743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:08.209801 748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:08.209775 749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:08.209739 751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:08.210898 743 server_base.cc:1034] running on GCE node
I20260812 16:38:08.211762 743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:08.212759 743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:08.214022 743 hybrid_clock.cc:648] HybridClock initialized: now 1786552688213941 us; error 54 us; skew 500 ppm
I20260812 16:38:08.214345 743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:08 dist-test-slave-f7th krb5kdc[625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552688, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:08.222860 743 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:38:08.224546 743 webserver.cc:460] Webserver started at http://127.31.61.129:35239/ using document root <none> and password file <none>
I20260812 16:38:08.225354 743 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:08.225449 743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:08.225752 743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:08.228848 743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "b8824b26e9b6480f9d326c2fd4a7450b"
format_stamp: "Formatted at 2026-08-12 16:38:08 on dist-test-slave-f7th"
server_key: "8bef40336a9e0e3a74d1b033f03387dc"
server_key_iv: "aae96fa9d5ca677e826290c2819710b7"
server_key_version: "encryptionkey@0"
I20260812 16:38:08.229684 743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "b8824b26e9b6480f9d326c2fd4a7450b"
format_stamp: "Formatted at 2026-08-12 16:38:08 on dist-test-slave-f7th"
server_key: "8bef40336a9e0e3a74d1b033f03387dc"
server_key_iv: "aae96fa9d5ca677e826290c2819710b7"
server_key_version: "encryptionkey@0"
I20260812 16:38:08.235217 743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:38:08.240010 758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:08.241562 743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 16:38:08.241750 743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "b8824b26e9b6480f9d326c2fd4a7450b"
format_stamp: "Formatted at 2026-08-12 16:38:08 on dist-test-slave-f7th"
server_key: "8bef40336a9e0e3a74d1b033f03387dc"
server_key_iv: "aae96fa9d5ca677e826290c2819710b7"
server_key_version: "encryptionkey@0"
I20260812 16:38:08.241993 743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:08.281215 743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:08.285950 743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:08.286222 743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:08.289382 743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:08.289490 743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:08.289628 743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:08.289688 743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:38:08 dist-test-slave-f7th krb5kdc[625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552688, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:08.315742 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:08.290882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45657 (local address 127.31.61.129:50217)
0812 16:38:08.291796 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:08.291853 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:08.294164 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:08.295158 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:08.295171 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:08.296222 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:08.297187 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:08.297217 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.298442 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.298448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:08.300001 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:08.300016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.301225 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.301230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:08.301527 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:08.302378 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:08.302409 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:08.306058 (+ 3649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:08.310070 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:08.310097 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:08.310121 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:08.310817 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:08.311357 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:08.311363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:08.311368 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:08.311711 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:08.312468 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:08.312481 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:08.313039 (+ 558us) client_negotiation.cc:769] Sending connection context
0812 16:38:08.313425 (+ 386us) client_negotiation.cc:240] Negotiation successful
0812 16:38:08.314044 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":366,"threads_started":1}
I20260812 16:38:08.316345 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:08.290170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:50217 (local address 127.31.61.190:45657)
0812 16:38:08.290651 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:08.290658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:08.292005 (+ 1347us) server_negotiation.cc:407] Connection header received
0812 16:38:08.294505 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:08.294512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:08.294613 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:08.294727 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:08.297392 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.298273 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.300224 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.300759 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.300891 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:08.306411 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:08.306420 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:08.306425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:08.306472 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:08.309862 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:08.311010 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:08.311016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:08.311020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:08.311191 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:08.311916 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:08.311922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:08.311925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:08.312327 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:08.312484 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:08.315809 (+ 3325us) server_negotiation.cc:299] Negotiation successful
0812 16:38:08.316001 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260812 16:38:08.337517 743 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:41603
I20260812 16:38:08.337476 881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:41603 every 8 connection(s)
I20260812 16:38:08.338929 743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:08.339573 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:08.324291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:58063 (local address 127.31.61.190:45657)
0812 16:38:08.324503 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:08.324510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:08.325003 (+ 493us) server_negotiation.cc:407] Connection header received
0812 16:38:08.325244 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:08.325251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:08.325343 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:08.325463 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:08.326945 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.328067 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.329875 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.330546 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.330745 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:08.332898 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:08.332904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:08.332910 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:08.332952 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:08.336490 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:08.337445 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:08.337452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:08.337454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:08.337604 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:08.338014 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:08.338020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:08.338024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:08.338332 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:08.338532 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:08.338965 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 16:38:08.339235 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 16:38:08.339679 847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:08.324259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45657 (local address 127.31.61.129:58063)
0812 16:38:08.324887 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:08.324910 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:08.325092 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:08.325524 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:08.325531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:08.326208 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:08.326696 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:08.326712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.328241 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.328245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:08.329672 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:08.329691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.330809 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.330819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:08.331038 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:08.331851 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:08.331880 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:08.332664 (+ 784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:08.336648 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:08.336654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:08.336661 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:08.337294 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:08.337727 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:08.337733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:08.337737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:08.337904 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:08.338497 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:08.338501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:08.338713 (+ 212us) client_negotiation.cc:769] Sending connection context
0812 16:38:08.338835 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 16:38:08.339205 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":333,"threads_started":1}
I20260812 16:38:08.341634 882 heartbeater.cc:347] Connected to a master server at 127.31.61.190:45657
I20260812 16:38:08.343132 882 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:08.343981 882 heartbeater.cc:511] Master 127.31.61.190:45657 requested a full tablet report, sending...
I20260812 16:38:08.346065 672 ts_manager.cc:194] Registered new tserver with Master: b8824b26e9b6480f9d326c2fd4a7450b (127.31.61.129:41603)
I20260812 16:38:08.347568 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 743
I20260812 16:38:08.347643 672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:50217
I20260812 16:38:08.347849 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:08.348487 31990 external_mini_cluster.cc:1428] Setting key a1c56a1940b424105efb9a19da19adf6
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:38:08.391144 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45657
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:08.719961 886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:08.720271 886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:08.720438 886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:08.728112 886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:08.728271 886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:08.728370 886 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:38:08.728453 886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:08.738745 886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.61.190:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:08.740202 886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:08.741554 886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:08.755693 891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:08.755628 892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:08.755580 894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:08.756719 886 server_base.cc:1034] running on GCE node
I20260812 16:38:08.757380 886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:08.758358 886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:08.759642 886 hybrid_clock.cc:648] HybridClock initialized: now 1786552688759524 us; error 54 us; skew 500 ppm
I20260812 16:38:08.760030 886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:08 dist-test-slave-f7th krb5kdc[625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552688, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:08.763765 886 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:38:08.765539 886 webserver.cc:460] Webserver started at http://127.31.61.130:34549/ using document root <none> and password file <none>
I20260812 16:38:08.766291 886 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:08.766399 886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:08.766734 886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:08.769976 886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "608afbc6433d4c9b93b0cb85484f6afa"
format_stamp: "Formatted at 2026-08-12 16:38:08 on dist-test-slave-f7th"
server_key: "553847a59250bb7c7f4c14ce7740947c"
server_key_iv: "541899503bdaf7aa4fcd771986909aa1"
server_key_version: "encryptionkey@0"
I20260812 16:38:08.770835 886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "608afbc6433d4c9b93b0cb85484f6afa"
format_stamp: "Formatted at 2026-08-12 16:38:08 on dist-test-slave-f7th"
server_key: "553847a59250bb7c7f4c14ce7740947c"
server_key_iv: "541899503bdaf7aa4fcd771986909aa1"
server_key_version: "encryptionkey@0"
I20260812 16:38:08.776918 886 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 16:38:08.780683 901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:08.782424 886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:38:08.782601 886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "608afbc6433d4c9b93b0cb85484f6afa"
format_stamp: "Formatted at 2026-08-12 16:38:08 on dist-test-slave-f7th"
server_key: "553847a59250bb7c7f4c14ce7740947c"
server_key_iv: "541899503bdaf7aa4fcd771986909aa1"
server_key_version: "encryptionkey@0"
I20260812 16:38:08.782841 886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:08.800436 886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:08.805081 886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:08.805331 886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:08.808144 886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:08.808236 886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:08.808385 886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:08.808462 886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:38:08 dist-test-slave-f7th krb5kdc[625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552688, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:08.831283 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:08.808625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:45841 (local address 127.31.61.190:45657)
0812 16:38:08.808822 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:08.808828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:08.810270 (+ 1442us) server_negotiation.cc:407] Connection header received
0812 16:38:08.812432 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:08.812439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:08.812541 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:08.812666 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:08.814987 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.816175 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.819180 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.819901 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.820040 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:08.824406 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:08.824412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:08.824416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:08.824452 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:08.827374 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:08.828493 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:08.828500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:08.828503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:08.828649 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:08.829240 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:08.829248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:08.829252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:08.829670 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:08.829855 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:08.830608 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 16:38:08.830879 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 16:38:08.832336 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:08.809206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45657 (local address 127.31.61.130:45841)
0812 16:38:08.810076 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:08.810134 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:08.812114 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:08.813067 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:08.813075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:08.813842 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:08.814716 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:08.814741 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.816422 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.816431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:08.818958 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:08.818979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.820166 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.820172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:08.820357 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:08.821126 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:08.821167 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:08.824173 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:08.827527 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:08.827538 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:08.827558 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:08.828344 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:08.828770 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:08.828774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:08.828780 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:08.829095 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:08.829843 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:08.829850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:08.830236 (+ 386us) client_negotiation.cc:769] Sending connection context
0812 16:38:08.830471 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 16:38:08.830894 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":375,"threads_started":1}
I20260812 16:38:08.854357 886 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:33181
I20260812 16:38:08.854389 1023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:33181 every 8 connection(s)
I20260812 16:38:08.855893 886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:08.856906 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:08.840573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:36857 (local address 127.31.61.190:45657)
0812 16:38:08.840779 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:08.840785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:08.840977 (+ 192us) server_negotiation.cc:407] Connection header received
0812 16:38:08.841200 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:08.841207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:08.841298 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:08.841422 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:08.842464 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.843712 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.846326 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.846986 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.847136 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:08.850383 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:08.850391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:08.850396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:08.850443 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:08.853815 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:08.854788 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:08.854793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:08.854796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:08.854949 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:08.855330 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:08.855334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:08.855336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:08.855735 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:08.855851 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:08.856350 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 16:38:08.856587 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:38:08.857177 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:08.840013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45657 (local address 127.31.61.130:36857)
0812 16:38:08.840817 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:08.840856 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:08.841040 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:08.841481 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:08.841487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:08.841965 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:08.842305 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:08.842315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.843935 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.843942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:08.846130 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:08.846148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:08.847248 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:08.847256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:08.847475 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:08.848298 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:08.848330 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:08.850140 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:08.854018 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:08.854024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:08.854028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:08.854641 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:08.855067 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:08.855072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:08.855075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:08.855229 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:08.855851 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:08.855855 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:08.856078 (+ 223us) client_negotiation.cc:769] Sending connection context
0812 16:38:08.856250 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 16:38:08.856652 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":762,"threads_started":1}
I20260812 16:38:08.858433 1024 heartbeater.cc:347] Connected to a master server at 127.31.61.190:45657
I20260812 16:38:08.858656 1024 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:08.859522 1024 heartbeater.cc:511] Master 127.31.61.190:45657 requested a full tablet report, sending...
I20260812 16:38:08.860029 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 886
I20260812 16:38:08.860253 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:08.861120 31990 external_mini_cluster.cc:1428] Setting key 7f126d8fb87a915655663ee45d6abe56
I20260812 16:38:08.861233 672 ts_manager.cc:194] Registered new tserver with Master: 608afbc6433d4c9b93b0cb85484f6afa (127.31.61.130:33181)
I20260812 16:38:08.862046 672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:45841
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:38:08.902832 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45657
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:09.234963 1028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:09.235323 1028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:09.235457 1028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:09.243868 1028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:09.244001 1028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:09.244102 1028 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:38:09.244266 1028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:09.255815 1028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.61.190:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:09.257261 1028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:09.258621 1028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:09.273696 1034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:09.273804 1028 server_base.cc:1034] running on GCE node
W20260812 16:38:09.273696 1036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:09.273748 1033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:09.274765 1028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:09.275831 1028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:09.277159 1028 hybrid_clock.cc:648] HybridClock initialized: now 1786552689277042 us; error 42 us; skew 500 ppm
I20260812 16:38:09.277498 1028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:09 dist-test-slave-f7th krb5kdc[625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552689, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:09.280994 1028 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:38:09.283136 1028 webserver.cc:460] Webserver started at http://127.31.61.131:42411/ using document root <none> and password file <none>
I20260812 16:38:09.284147 1028 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:09.284518 1028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:09.284972 1028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:09.288156 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6"
format_stamp: "Formatted at 2026-08-12 16:38:09 on dist-test-slave-f7th"
server_key: "4c0149ee4cb4f47a8473e7094ee595f9"
server_key_iv: "50d607a0adf26ddd240da8fbc91e3e48"
server_key_version: "encryptionkey@0"
I20260812 16:38:09.288972 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6"
format_stamp: "Formatted at 2026-08-12 16:38:09 on dist-test-slave-f7th"
server_key: "4c0149ee4cb4f47a8473e7094ee595f9"
server_key_iv: "50d607a0adf26ddd240da8fbc91e3e48"
server_key_version: "encryptionkey@0"
I20260812 16:38:09.294889 1028 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 16:38:09.298659 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:09.300211 1028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:09.300380 1028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6"
format_stamp: "Formatted at 2026-08-12 16:38:09 on dist-test-slave-f7th"
server_key: "4c0149ee4cb4f47a8473e7094ee595f9"
server_key_iv: "50d607a0adf26ddd240da8fbc91e3e48"
server_key_version: "encryptionkey@0"
I20260812 16:38:09.300595 1028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:09.326433 1028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:09.331153 1028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:09.331450 1028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:09.334482 1028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:09.334596 1028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:09.334728 1028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:09.334794 1028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:38:09 dist-test-slave-f7th krb5kdc[625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552689, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:09.350845 882 heartbeater.cc:503] Master 127.31.61.190:45657 was elected leader, sending a full tablet report...
I20260812 16:38:09.359906 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.335223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:44787 (local address 127.31.61.190:45657)
0812 16:38:09.335657 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:09.335662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:09.337313 (+ 1651us) server_negotiation.cc:407] Connection header received
0812 16:38:09.340473 (+ 3160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:09.340481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:09.340582 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:09.340657 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:09.343310 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.344350 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.346602 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.347033 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.347167 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.352507 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:09.352514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:09.352518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:09.352560 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:09.355962 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:09.356989 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:09.356996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:09.357000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:09.357158 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:09.357769 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:09.357775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:09.357778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:09.358135 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:09.358251 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:09.359390 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0812 16:38:09.359628 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260812 16:38:09.361670 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.336043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45657 (local address 127.31.61.131:44787)
0812 16:38:09.337141 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:09.337196 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:09.340095 (+ 2899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:09.341210 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:09.341221 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:09.342049 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:09.343089 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.343120 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.344508 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.344513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.346405 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.346419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.347320 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.347329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.347589 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.348477 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:09.348511 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:09.351852 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:09.356103 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:09.356113 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:09.356136 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:09.356813 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:09.357270 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:09.357279 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:09.357283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:09.357602 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:09.358276 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:09.358287 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:09.358881 (+ 594us) client_negotiation.cc:769] Sending connection context
0812 16:38:09.359247 (+ 366us) client_negotiation.cc:240] Negotiation successful
0812 16:38:09.359921 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":521,"threads_started":1}
I20260812 16:38:09.382531 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.367701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:43145 (local address 127.31.61.190:45657)
0812 16:38:09.367923 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:09.367930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:09.368818 (+ 888us) server_negotiation.cc:407] Connection header received
0812 16:38:09.368996 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:09.369000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:09.369064 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:09.369148 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:09.370125 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.370897 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.372555 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.373119 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.373240 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.375787 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:09.375795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:09.375801 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:09.375848 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:09.379323 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:09.380219 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:09.380226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:09.380229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:09.380404 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:09.380812 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:09.380819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:09.380822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:09.381240 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:09.381409 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:09.381933 (+ 524us) server_negotiation.cc:299] Negotiation successful
0812 16:38:09.382159 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:38:09.382673 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.368106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45657 (local address 127.31.61.131:43145)
0812 16:38:09.368711 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:09.368729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:09.368872 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:09.369189 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:09.369194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:09.369689 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:09.369977 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.369987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.371036 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.371042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.372410 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.372427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.373343 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.373353 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.373590 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.374375 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:09.374403 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:09.375546 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:09.379506 (+ 3960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:09.379510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:09.379513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:09.380063 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:09.380536 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:09.380541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:09.380546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:09.380684 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:09.381360 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:09.381368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:09.381596 (+ 228us) client_negotiation.cc:769] Sending connection context
0812 16:38:09.381770 (+ 174us) client_negotiation.cc:240] Negotiation successful
0812 16:38:09.382159 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":356,"threads_started":1}
I20260812 16:38:09.393429 1028 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:36177
I20260812 16:38:09.393479 1163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:36177 every 8 connection(s)
I20260812 16:38:09.394639 1028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:09.396764 1164 heartbeater.cc:347] Connected to a master server at 127.31.61.190:45657
I20260812 16:38:09.397004 1164 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:09.397666 1164 heartbeater.cc:511] Master 127.31.61.190:45657 requested a full tablet report, sending...
I20260812 16:38:09.399092 671 ts_manager.cc:194] Registered new tserver with Master: a3cd30cbbb7e4644b5b5749aa48b8ce6 (127.31.61.131:36177)
I20260812 16:38:09.400044 671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:44787
I20260812 16:38:09.403241 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 1028
I20260812 16:38:09.403427 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:09.403946 31990 external_mini_cluster.cc:1428] Setting key 662b63c4669ede50ae59cd2364cfbfd3
I20260812 16:38:09.406980 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:38:09 dist-test-slave-f7th krb5kdc[625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:38:09 dist-test-slave-f7th krb5kdc[625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:09.463069 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.446320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.31.61.190:45657)
0812 16:38:09.446557 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:09.446563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:09.447032 (+ 469us) server_negotiation.cc:407] Connection header received
0812 16:38:09.447257 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:09.447263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:09.447346 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:09.447456 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:09.449229 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.449985 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.451805 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.452228 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.452351 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.457073 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:09.457080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:09.457083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:09.457130 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:09.459941 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:09.460855 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:09.460864 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:09.460868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:09.461025 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:09.461471 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:09.461478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:09.461481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:09.461804 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:09.462034 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:09.462478 (+ 444us) server_negotiation.cc:299] Negotiation successful
0812 16:38:09.462687 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 16:38:09.467739 671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:09.474828 671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:09.500469 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.487799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.31.61.129:41603)
0812 16:38:09.490272 (+ 2473us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:09.490283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:09.490317 (+ 34us) server_negotiation.cc:407] Connection header received
0812 16:38:09.490417 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:09.490424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:09.490753 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:09.490963 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:09.493080 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.494735 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.498004 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.499131 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.499270 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.499497 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:09.500042 (+ 545us) server_negotiation.cc:299] Negotiation successful
0812 16:38:09.500204 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1320,"thread_start_us":1190,"threads_started":1}
I20260812 16:38:09.500519 1179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.487537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:41603 (local address 127.0.0.1:41536)
0812 16:38:09.489742 (+ 2205us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:09.489762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:09.489899 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:09.491253 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:09.491262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:09.491299 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:09.491785 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.491810 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.494963 (+ 3153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.494969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.497805 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.497818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.499673 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.499679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.499855 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.499879 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 16:38:09.499943 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 16:38:09.500098 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2072,"mutex_wait_us":4,"thread_start_us":1907,"threads_started":1}
I20260812 16:38:09.500597 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.487834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.31.61.131:36177)
0812 16:38:09.490642 (+ 2808us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:09.490652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:09.490687 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:38:09.490800 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:09.490807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:09.491036 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:09.491233 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:09.492353 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.494222 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.497735 (+ 3513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.499166 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.499288 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.499535 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:09.500096 (+ 561us) server_negotiation.cc:299] Negotiation successful
0812 16:38:09.500264 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2630,"thread_start_us":2448,"threads_started":1}
I20260812 16:38:09.501308 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.487489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:36177 (local address 127.0.0.1:60184)
0812 16:38:09.488417 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:09.488481 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:09.488663 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:09.491549 (+ 2886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:09.491553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:09.491582 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:09.492103 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.492121 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.494430 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.494437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.497521 (+ 3084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.497534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.499413 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.499420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.499683 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.499727 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 16:38:09.499846 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 16:38:09.500942 (+ 1096us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":1375,"thread_start_us":484,"threads_started":1}
I20260812 16:38:09.504732 810 tablet_service.cc:1467] Processing CreateTablet for tablet c0b06da21bb748aca4456279a2185543 (DEFAULT_TABLE table=test-table [id=8cfbff33101143809c7bb0ed8bf3dbaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:09.504771 1093 tablet_service.cc:1467] Processing CreateTablet for tablet c0b06da21bb748aca4456279a2185543 (DEFAULT_TABLE table=test-table [id=8cfbff33101143809c7bb0ed8bf3dbaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:09.506151 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.490355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:33181 (local address 127.0.0.1:49298)
0812 16:38:09.497141 (+ 6786us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:09.497161 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:09.497284 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:09.499117 (+ 1833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:09.499121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:09.499149 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:09.499654 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.499667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.501387 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.501391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.503396 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.503412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.505306 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.505315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.505559 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.505594 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:38:09.505699 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 16:38:09.505830 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6645,"thread_start_us":6470,"threads_started":1}
I20260812 16:38:09.506211 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.490862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.31.61.130:33181)
0812 16:38:09.498444 (+ 7582us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:09.498454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:09.498486 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:38:09.498583 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:09.498592 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:09.498874 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:09.499107 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:09.499850 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.501218 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.503627 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.504992 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.505147 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.505350 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:09.505765 (+ 415us) server_negotiation.cc:299] Negotiation successful
0812 16:38:09.505948 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7413,"thread_start_us":7321,"threads_started":1}
I20260812 16:38:09.506242 810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b06da21bb748aca4456279a2185543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:09.506256 1093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b06da21bb748aca4456279a2185543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:09.510758 952 tablet_service.cc:1467] Processing CreateTablet for tablet c0b06da21bb748aca4456279a2185543 (DEFAULT_TABLE table=test-table [id=8cfbff33101143809c7bb0ed8bf3dbaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:09.512329 952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b06da21bb748aca4456279a2185543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:09.518153 1183 tablet_bootstrap.cc:492] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: Bootstrap starting.
I20260812 16:38:09.522059 1183 tablet_bootstrap.cc:654] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:09.522219 1184 tablet_bootstrap.cc:492] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Bootstrap starting.
I20260812 16:38:09.523707 1183 log.cc:824] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:09.526984 1184 tablet_bootstrap.cc:654] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:09.528322 1186 tablet_bootstrap.cc:492] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: Bootstrap starting.
I20260812 16:38:09.528950 1184 log.cc:824] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:09.530176 1183 tablet_bootstrap.cc:492] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: No bootstrap required, opened a new log
I20260812 16:38:09.530552 1183 ts_tablet_manager.cc:1397] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260812 16:38:09.532934 1186 tablet_bootstrap.cc:654] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:09.534749 1186 log.cc:824] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:09.535128 1184 tablet_bootstrap.cc:492] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: No bootstrap required, opened a new log
I20260812 16:38:09.535502 1184 ts_tablet_manager.cc:1397] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260812 16:38:09.540359 1183 raft_consensus.cc:348] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } }
I20260812 16:38:09.541081 1183 raft_consensus.cc:374] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:09.541200 1183 raft_consensus.cc:729] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8824b26e9b6480f9d326c2fd4a7450b, State: Initialized, Role: FOLLOWER
I20260812 16:38:09.541893 1183 consensus_queue.cc:260] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } }
I20260812 16:38:09.543529 1183 ts_tablet_manager.cc:1428] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260812 16:38:09.545089 1186 tablet_bootstrap.cc:492] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: No bootstrap required, opened a new log
I20260812 16:38:09.545575 1186 ts_tablet_manager.cc:1397] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.003s
I20260812 16:38:09.547917 1184 raft_consensus.cc:348] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } }
I20260812 16:38:09.548430 1184 raft_consensus.cc:374] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:09.548549 1184 raft_consensus.cc:729] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cd30cbbb7e4644b5b5749aa48b8ce6, State: Initialized, Role: FOLLOWER
I20260812 16:38:09.549136 1184 consensus_queue.cc:260] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } }
I20260812 16:38:09.550518 1164 heartbeater.cc:503] Master 127.31.61.190:45657 was elected leader, sending a full tablet report...
I20260812 16:38:09.551028 1184 ts_tablet_manager.cc:1428] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260812 16:38:09.556737 1186 raft_consensus.cc:348] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } }
I20260812 16:38:09.557192 1186 raft_consensus.cc:374] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:09.557365 1186 raft_consensus.cc:729] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608afbc6433d4c9b93b0cb85484f6afa, State: Initialized, Role: FOLLOWER
I20260812 16:38:09.558005 1186 consensus_queue.cc:260] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } }
I20260812 16:38:09.559374 1024 heartbeater.cc:503] Master 127.31.61.190:45657 was elected leader, sending a full tablet report...
I20260812 16:38:09.559943 1186 ts_tablet_manager.cc:1428] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260812 16:38:09.592969 883 tablet.cc:2369] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:38:09.609328 1025 tablet.cc:2369] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:38:09.646327 1165 tablet.cc:2369] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:38:09.763698 1192 raft_consensus.cc:482] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:09.764107 1192 raft_consensus.cc:504] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } }
I20260812 16:38:09.766096 1192 leader_election.cc:288] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cd30cbbb7e4644b5b5749aa48b8ce6 (127.31.61.131:36177), b8824b26e9b6480f9d326c2fd4a7450b (127.31.61.129:41603)
I20260812 16:38:09.774739 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.766706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:45041 (local address 127.31.61.129:41603)
0812 16:38:09.767007 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:09.767013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:09.768225 (+ 1212us) server_negotiation.cc:407] Connection header received
0812 16:38:09.768446 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:09.768452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:09.768541 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:09.768620 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:09.769393 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.770335 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.772957 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.773724 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.773854 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.773932 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:09.774290 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 16:38:09.774427 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 16:38:09.774767 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.766747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:41603 (local address 127.31.61.130:45041)
0812 16:38:09.768081 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:09.768106 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:09.768303 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:09.768658 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:09.768663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:09.768688 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:09.769236 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.769247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.770465 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.770471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.772798 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.772811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.773964 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.773971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.774132 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.774156 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 16:38:09.774230 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:38:09.774325 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":30,"thread_start_us":656,"threads_started":1}
I20260812 16:38:09.774793 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.766838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:40843 (local address 127.31.61.131:36177)
0812 16:38:09.767149 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:09.767157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:09.768425 (+ 1268us) server_negotiation.cc:407] Connection header received
0812 16:38:09.768573 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:09.768577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:09.768648 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:09.768739 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:09.769830 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.770854 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.772917 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.773761 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.773923 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.773992 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:09.774443 (+ 451us) server_negotiation.cc:299] Negotiation successful
0812 16:38:09.774571 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260812 16:38:09.775019 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.767070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:36177 (local address 127.31.61.130:40843)
0812 16:38:09.768315 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:09.768329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:09.768466 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:09.768770 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:09.768773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:09.768807 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:09.769658 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.769675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.771020 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.771024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.772763 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:09.772776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.774095 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.774100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:09.774271 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.774307 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 16:38:09.774394 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 16:38:09.774515 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":39,"thread_start_us":792,"threads_started":1}
I20260812 16:38:09.776219 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b06da21bb748aca4456279a2185543" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" is_pre_election: true
I20260812 16:38:09.776270 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b06da21bb748aca4456279a2185543" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" is_pre_election: true
I20260812 16:38:09.776885 834 raft_consensus.cc:2454] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 0.
I20260812 16:38:09.776881 1119 raft_consensus.cc:2454] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 0.
I20260812 16:38:09.777573 903 leader_election.cc:302] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608afbc6433d4c9b93b0cb85484f6afa, a3cd30cbbb7e4644b5b5749aa48b8ce6; no voters:
I20260812 16:38:09.778093 1192 raft_consensus.cc:2781] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:09.778318 1192 raft_consensus.cc:482] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:09.778513 1192 raft_consensus.cc:3037] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:09.781116 1192 raft_consensus.cc:504] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } }
I20260812 16:38:09.781842 1192 leader_election.cc:288] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 election: Requested vote from peers a3cd30cbbb7e4644b5b5749aa48b8ce6 (127.31.61.131:36177), b8824b26e9b6480f9d326c2fd4a7450b (127.31.61.129:41603)
I20260812 16:38:09.782334 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b06da21bb748aca4456279a2185543" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8824b26e9b6480f9d326c2fd4a7450b"
I20260812 16:38:09.782436 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b06da21bb748aca4456279a2185543" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6"
I20260812 16:38:09.782673 1119 raft_consensus.cc:3037] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:09.782719 834 raft_consensus.cc:3037] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:09.785207 1119 raft_consensus.cc:2454] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 1.
I20260812 16:38:09.785745 903 leader_election.cc:302] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608afbc6433d4c9b93b0cb85484f6afa, a3cd30cbbb7e4644b5b5749aa48b8ce6; no voters:
I20260812 16:38:09.786018 834 raft_consensus.cc:2454] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 1.
I20260812 16:38:09.786243 1192 raft_consensus.cc:2781] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:09.787305 1192 raft_consensus.cc:686] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 LEADER]: Becoming Leader. State: Replica: 608afbc6433d4c9b93b0cb85484f6afa, State: Running, Role: LEADER
I20260812 16:38:09.787979 1192 consensus_queue.cc:237] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } }
I20260812 16:38:09.792920 671 catalog_manager.cc:5495] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa reported cstate change: term changed from 0 to 1, leader changed from <none> to 608afbc6433d4c9b93b0cb85484f6afa (127.31.61.130). New cstate: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } health_report { overall_health: HEALTHY } } }
I20260812 16:38:09.888202 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:09.881627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.31.61.130:33181)
0812 16:38:09.881944 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:09.881950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:09.881978 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:38:09.882259 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:09.882263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:09.882362 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:09.882487 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:09.882931 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.883977 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.885543 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:09.886007 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:09.886135 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:09.886717 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:09.887414 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:09.887577 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:09.887748 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 16:38:09.887843 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260812 16:38:09.899212 834 raft_consensus.cc:1261] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Refusing update from remote peer 608afbc6433d4c9b93b0cb85484f6afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:09.900265 1119 raft_consensus.cc:1261] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 1 FOLLOWER]: Refusing update from remote peer 608afbc6433d4c9b93b0cb85484f6afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:09.900753 1195 consensus_queue.cc:1035] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:09.901392 1192 consensus_queue.cc:1035] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:09.915916 1200 mvcc.cc:205] Tried to move back new op lower bound from 7317719817810837504 to 7317719817376354304. Current Snapshot: MvccSnapshot[applied={T|T < 7317719817810837504}]
I20260812 16:38:09.919234 1198 mvcc.cc:205] Tried to move back new op lower bound from 7317719817810837504 to 7317719817376354304. Current Snapshot: MvccSnapshot[applied={T|T < 7317719817810837504 or (T in {7317719817810837504})}]
I20260812 16:38:09.939307 671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:09.939733 671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:09.944746 671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f: Sending DeleteTablet for 3 replicas of tablet c0b06da21bb748aca4456279a2185543
I20260812 16:38:09.946424 810 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b06da21bb748aca4456279a2185543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:09 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:41536
I20260812 16:38:09.946441 1093 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b06da21bb748aca4456279a2185543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:09 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:60184
I20260812 16:38:09.946601 952 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b06da21bb748aca4456279a2185543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:09 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:49298
I20260812 16:38:09.955489 671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:09.956326 671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:09.957558 1210 tablet_replica.cc:330] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: stopping tablet replica
I20260812 16:38:09.959381 1210 raft_consensus.cc:2229] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:09.959570 1208 tablet_replica.cc:330] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: stopping tablet replica
I20260812 16:38:09.960172 1208 raft_consensus.cc:2229] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:09.960883 1210 raft_consensus.cc:2258] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:09.961014 1208 raft_consensus.cc:2258] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:09.964072 1210 ts_tablet_manager.cc:1905] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:09.965447 1208 ts_tablet_manager.cc:1905] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:09.966256 809 tablet_service.cc:1467] Processing CreateTablet for tablet 9f795d0adfb24fb9976a6b8e5d81d892 (DEFAULT_TABLE table=test-table [id=54dcd5c1f5e749f9a2043c8f4766ad3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:09.966913 809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f795d0adfb24fb9976a6b8e5d81d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:09.966948 952 tablet_service.cc:1467] Processing CreateTablet for tablet 9f795d0adfb24fb9976a6b8e5d81d892 (DEFAULT_TABLE table=test-table [id=54dcd5c1f5e749f9a2043c8f4766ad3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:09.972570 1093 tablet_service.cc:1467] Processing CreateTablet for tablet 9f795d0adfb24fb9976a6b8e5d81d892 (DEFAULT_TABLE table=test-table [id=54dcd5c1f5e749f9a2043c8f4766ad3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:09.972985 1210 ts_tablet_manager.cc:1918] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:09.973204 1093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f795d0adfb24fb9976a6b8e5d81d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:09.973320 1210 log.cc:1196] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/c0b06da21bb748aca4456279a2185543
I20260812 16:38:09.973599 1183 tablet_bootstrap.cc:492] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: Bootstrap starting.
I20260812 16:38:09.974257 1210 ts_tablet_manager.cc:1939] T c0b06da21bb748aca4456279a2185543 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Deleting consensus metadata
I20260812 16:38:09.975718 1209 tablet_replica.cc:330] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: stopping tablet replica
I20260812 16:38:09.976233 1209 raft_consensus.cc:2229] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:09.976830 656 catalog_manager.cc:4841] TS a3cd30cbbb7e4644b5b5749aa48b8ce6 (127.31.61.131:36177): tablet c0b06da21bb748aca4456279a2185543 (table test-table [id=8cfbff33101143809c7bb0ed8bf3dbaa]) successfully deleted
I20260812 16:38:09.977138 1183 tablet_bootstrap.cc:654] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:09.977115 1209 raft_consensus.cc:2258] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:09.977392 952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f795d0adfb24fb9976a6b8e5d81d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:09.980371 1183 tablet_bootstrap.cc:492] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: No bootstrap required, opened a new log
I20260812 16:38:09.980528 1183 ts_tablet_manager.cc:1397] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 16:38:09.981572 1209 ts_tablet_manager.cc:1905] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:09.983278 1183 raft_consensus.cc:348] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:09.983558 1183 raft_consensus.cc:374] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:09.985239 1186 tablet_bootstrap.cc:492] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: Bootstrap starting.
I20260812 16:38:09.987308 1186 tablet_bootstrap.cc:654] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:09.989611 1184 tablet_bootstrap.cc:492] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Bootstrap starting.
I20260812 16:38:09.991492 1184 tablet_bootstrap.cc:654] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:09.991829 1209 ts_tablet_manager.cc:1918] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:09.992285 1209 log.cc:1196] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/c0b06da21bb748aca4456279a2185543
I20260812 16:38:09.993155 1209 ts_tablet_manager.cc:1939] T c0b06da21bb748aca4456279a2185543 P 608afbc6433d4c9b93b0cb85484f6afa: Deleting consensus metadata
I20260812 16:38:09.994688 1208 ts_tablet_manager.cc:1918] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:09.994870 1208 log.cc:1196] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/c0b06da21bb748aca4456279a2185543
I20260812 16:38:09.995523 1208 ts_tablet_manager.cc:1939] T c0b06da21bb748aca4456279a2185543 P b8824b26e9b6480f9d326c2fd4a7450b: Deleting consensus metadata
I20260812 16:38:09.995540 658 catalog_manager.cc:4841] TS 608afbc6433d4c9b93b0cb85484f6afa (127.31.61.130:33181): tablet c0b06da21bb748aca4456279a2185543 (table test-table [id=8cfbff33101143809c7bb0ed8bf3dbaa]) successfully deleted
I20260812 16:38:09.997156 1183 raft_consensus.cc:729] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8824b26e9b6480f9d326c2fd4a7450b, State: Initialized, Role: FOLLOWER
I20260812 16:38:09.997238 657 catalog_manager.cc:4841] TS b8824b26e9b6480f9d326c2fd4a7450b (127.31.61.129:41603): tablet c0b06da21bb748aca4456279a2185543 (table test-table [id=8cfbff33101143809c7bb0ed8bf3dbaa]) successfully deleted
I20260812 16:38:09.997615 1183 consensus_queue.cc:260] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:09.999512 1184 tablet_bootstrap.cc:492] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: No bootstrap required, opened a new log
I20260812 16:38:09.999832 1183 ts_tablet_manager.cc:1428] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20260812 16:38:09.999852 1184 ts_tablet_manager.cc:1397] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20260812 16:38:10.001359 1184 raft_consensus.cc:348] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.001652 1184 raft_consensus.cc:374] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:10.001752 1184 raft_consensus.cc:729] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cd30cbbb7e4644b5b5749aa48b8ce6, State: Initialized, Role: FOLLOWER
I20260812 16:38:10.002025 1184 consensus_queue.cc:260] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.002828 1184 ts_tablet_manager.cc:1428] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 16:38:10.003844 1186 tablet_bootstrap.cc:492] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: No bootstrap required, opened a new log
I20260812 16:38:10.004014 1186 ts_tablet_manager.cc:1397] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.010s
I20260812 16:38:10.005409 1186 raft_consensus.cc:348] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.005653 1186 raft_consensus.cc:374] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:10.005765 1186 raft_consensus.cc:729] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608afbc6433d4c9b93b0cb85484f6afa, State: Initialized, Role: FOLLOWER
I20260812 16:38:10.006036 1186 consensus_queue.cc:260] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.007675 1186 ts_tablet_manager.cc:1428] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:10.081472 1192 raft_consensus.cc:482] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:10.081681 1192 raft_consensus.cc:504] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.082388 1192 leader_election.cc:288] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8824b26e9b6480f9d326c2fd4a7450b (127.31.61.129:41603), a3cd30cbbb7e4644b5b5749aa48b8ce6 (127.31.61.131:36177)
I20260812 16:38:10.083011 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f795d0adfb24fb9976a6b8e5d81d892" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" is_pre_election: true
I20260812 16:38:10.083271 1119 raft_consensus.cc:2454] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 0.
I20260812 16:38:10.083364 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f795d0adfb24fb9976a6b8e5d81d892" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" is_pre_election: true
I20260812 16:38:10.083643 834 raft_consensus.cc:2454] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 0.
I20260812 16:38:10.083868 903 leader_election.cc:302] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608afbc6433d4c9b93b0cb85484f6afa, a3cd30cbbb7e4644b5b5749aa48b8ce6; no voters:
I20260812 16:38:10.084311 1192 raft_consensus.cc:2781] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:10.084455 1192 raft_consensus.cc:482] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:10.084578 1192 raft_consensus.cc:3037] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:10.087252 1192 raft_consensus.cc:504] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.088294 1192 leader_election.cc:288] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 election: Requested vote from peers b8824b26e9b6480f9d326c2fd4a7450b (127.31.61.129:41603), a3cd30cbbb7e4644b5b5749aa48b8ce6 (127.31.61.131:36177)
I20260812 16:38:10.088428 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f795d0adfb24fb9976a6b8e5d81d892" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6"
I20260812 16:38:10.088510 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f795d0adfb24fb9976a6b8e5d81d892" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8824b26e9b6480f9d326c2fd4a7450b"
I20260812 16:38:10.088652 1119 raft_consensus.cc:3037] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:10.088762 834 raft_consensus.cc:3037] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:10.091073 1119 raft_consensus.cc:2454] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 1.
I20260812 16:38:10.091518 903 leader_election.cc:302] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608afbc6433d4c9b93b0cb85484f6afa, a3cd30cbbb7e4644b5b5749aa48b8ce6; no voters:
I20260812 16:38:10.091830 834 raft_consensus.cc:2454] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 1.
I20260812 16:38:10.091898 1192 raft_consensus.cc:2781] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:10.092286 1192 raft_consensus.cc:686] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 LEADER]: Becoming Leader. State: Replica: 608afbc6433d4c9b93b0cb85484f6afa, State: Running, Role: LEADER
I20260812 16:38:10.092604 1192 consensus_queue.cc:237] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.095575 672 catalog_manager.cc:5495] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa reported cstate change: term changed from 0 to 1, leader changed from <none> to 608afbc6433d4c9b93b0cb85484f6afa (127.31.61.130). New cstate: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } health_report { overall_health: UNKNOWN } } }
Aug 12 16:38:10 dist-test-slave-f7th krb5kdc[625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552687, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:10.157047 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.138422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45657 (local address 127.0.0.1:42526)
0812 16:38:10.139659 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:10.139680 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:10.139846 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:10.140215 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:10.140221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:10.141165 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:10.141751 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:10.141771 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.143479 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.143486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:10.145096 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:10.145113 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.146213 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.146221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:10.146393 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.147261 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:10.147288 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:10.150367 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:10.154002 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:10.154017 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:10.154027 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:10.154483 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:10.154890 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:10.154896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:10.154900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:10.155059 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:10.155690 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:10.155699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:10.155905 (+ 206us) client_negotiation.cc:769] Sending connection context
0812 16:38:10.156066 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 16:38:10.156389 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":884,"threads_started":1}
I20260812 16:38:10.157137 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.138393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.31.61.190:45657)
0812 16:38:10.139536 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:10.139542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:10.139767 (+ 225us) server_negotiation.cc:407] Connection header received
0812 16:38:10.139993 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:10.139997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:10.140064 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:10.140147 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:10.142183 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.143231 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.145267 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.146015 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.146117 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.150661 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:10.150668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:10.150671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:10.150706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:10.153850 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.154619 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.154624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.154626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.154770 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.155177 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.155181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.155183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.155526 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:10.155719 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:10.156248 (+ 529us) server_negotiation.cc:299] Negotiation successful
0812 16:38:10.156527 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":916,"thread_start_us":633,"threads_started":1}
I20260812 16:38:10.163667 672 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:42526:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 16:38:10.165024 672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:10.172467 1093 tablet_service.cc:1467] Processing CreateTablet for tablet dc8d12c6aa604a52825bad71508df775 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b172856533e242279b61ef7ca34a6d72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:38:10.172381 810 tablet_service.cc:1467] Processing CreateTablet for tablet dc8d12c6aa604a52825bad71508df775 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b172856533e242279b61ef7ca34a6d72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:38:10.172600 952 tablet_service.cc:1467] Processing CreateTablet for tablet dc8d12c6aa604a52825bad71508df775 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b172856533e242279b61ef7ca34a6d72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:38:10.173031 1093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc8d12c6aa604a52825bad71508df775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:10.173175 810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc8d12c6aa604a52825bad71508df775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:10.173190 952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc8d12c6aa604a52825bad71508df775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:10.183704 1183 tablet_bootstrap.cc:492] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: Bootstrap starting.
I20260812 16:38:10.185709 1184 tablet_bootstrap.cc:492] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Bootstrap starting.
I20260812 16:38:10.186182 1183 tablet_bootstrap.cc:654] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:10.186631 1186 tablet_bootstrap.cc:492] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: Bootstrap starting.
I20260812 16:38:10.187891 1184 tablet_bootstrap.cc:654] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:10.188949 1186 tablet_bootstrap.cc:654] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:10.198987 1184 tablet_bootstrap.cc:492] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: No bootstrap required, opened a new log
I20260812 16:38:10.199236 1184 ts_tablet_manager.cc:1397] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260812 16:38:10.199352 1183 tablet_bootstrap.cc:492] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: No bootstrap required, opened a new log
I20260812 16:38:10.199529 1183 ts_tablet_manager.cc:1397] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20260812 16:38:10.200407 1184 raft_consensus.cc:348] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.200522 1183 raft_consensus.cc:348] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.200693 1184 raft_consensus.cc:374] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:10.200740 1183 raft_consensus.cc:374] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:10.200778 1184 raft_consensus.cc:729] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cd30cbbb7e4644b5b5749aa48b8ce6, State: Initialized, Role: FOLLOWER
I20260812 16:38:10.200821 1183 raft_consensus.cc:729] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8824b26e9b6480f9d326c2fd4a7450b, State: Initialized, Role: FOLLOWER
I20260812 16:38:10.201114 1183 consensus_queue.cc:260] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.201193 1184 consensus_queue.cc:260] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.201866 1183 ts_tablet_manager.cc:1428] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 16:38:10.201722 1191 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.202069 1191 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:10.201880 1190 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.202879 1190 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:10.203003 1184 ts_tablet_manager.cc:1428] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260812 16:38:10.206300 1186 tablet_bootstrap.cc:492] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: No bootstrap required, opened a new log
I20260812 16:38:10.206498 1186 ts_tablet_manager.cc:1397] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.012s
I20260812 16:38:10.208024 1186 raft_consensus.cc:348] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.208457 1186 raft_consensus.cc:374] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:10.208598 1186 raft_consensus.cc:729] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608afbc6433d4c9b93b0cb85484f6afa, State: Initialized, Role: FOLLOWER
I20260812 16:38:10.208954 1186 consensus_queue.cc:260] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.209822 1186 ts_tablet_manager.cc:1428] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:10.209529 1192 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.210176 1192 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:10.519096 1192 raft_consensus.cc:482] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:10.519330 1192 raft_consensus.cc:504] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.520082 1192 leader_election.cc:288] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8824b26e9b6480f9d326c2fd4a7450b (127.31.61.129:41603), a3cd30cbbb7e4644b5b5749aa48b8ce6 (127.31.61.131:36177)
I20260812 16:38:10.520689 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8d12c6aa604a52825bad71508df775" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" is_pre_election: true
I20260812 16:38:10.520721 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8d12c6aa604a52825bad71508df775" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" is_pre_election: true
I20260812 16:38:10.520925 834 raft_consensus.cc:2454] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 0.
I20260812 16:38:10.520983 1119 raft_consensus.cc:2454] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 0.
I20260812 16:38:10.521371 904 leader_election.cc:302] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608afbc6433d4c9b93b0cb85484f6afa, b8824b26e9b6480f9d326c2fd4a7450b; no voters:
I20260812 16:38:10.521644 1192 raft_consensus.cc:2781] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:10.521734 1192 raft_consensus.cc:482] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:10.521840 1192 raft_consensus.cc:3037] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:10.523917 1192 raft_consensus.cc:504] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.524607 1192 leader_election.cc:288] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 election: Requested vote from peers b8824b26e9b6480f9d326c2fd4a7450b (127.31.61.129:41603), a3cd30cbbb7e4644b5b5749aa48b8ce6 (127.31.61.131:36177)
I20260812 16:38:10.525067 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8d12c6aa604a52825bad71508df775" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8824b26e9b6480f9d326c2fd4a7450b"
I20260812 16:38:10.525100 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8d12c6aa604a52825bad71508df775" candidate_uuid: "608afbc6433d4c9b93b0cb85484f6afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6"
I20260812 16:38:10.525287 834 raft_consensus.cc:3037] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:10.525290 1119 raft_consensus.cc:3037] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:10.527316 834 raft_consensus.cc:2454] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 1.
I20260812 16:38:10.527343 1119 raft_consensus.cc:2454] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608afbc6433d4c9b93b0cb85484f6afa in term 1.
I20260812 16:38:10.527760 904 leader_election.cc:302] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608afbc6433d4c9b93b0cb85484f6afa, b8824b26e9b6480f9d326c2fd4a7450b; no voters:
I20260812 16:38:10.528270 1192 raft_consensus.cc:2781] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:10.528445 1192 raft_consensus.cc:686] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 LEADER]: Becoming Leader. State: Replica: 608afbc6433d4c9b93b0cb85484f6afa, State: Running, Role: LEADER
I20260812 16:38:10.528699 1192 consensus_queue.cc:237] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } }
I20260812 16:38:10.530405 1203 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: TxnStatusTablet state changed. Reason: New leader 608afbc6433d4c9b93b0cb85484f6afa. Latest consensus state: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.530761 1203 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:38:10.531996 671 catalog_manager.cc:5495] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa reported cstate change: term changed from 0 to 1, leader changed from <none> to 608afbc6433d4c9b93b0cb85484f6afa (127.31.61.130). New cstate: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } health_report { overall_health: UNKNOWN } } }
I20260812 16:38:10.532642 1226 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 16:38:10.532929 1226 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 16:38:10.564640 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.559148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:33181 (local address 127.0.0.1:49304)
0812 16:38:10.559351 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:10.559367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:10.559503 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:10.560250 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:10.560255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:10.560268 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:38:10.560449 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:10.560458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.561562 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.561566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:10.562536 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:10.562557 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.563337 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.563344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:10.563540 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.563641 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:38:10.564066 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:38:10.564088 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 16:38:10.564162 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:38:10.564321 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 16:38:10.564692 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.559295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.31.61.130:33181)
0812 16:38:10.559926 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:10.559931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:10.559946 (+ 15us) server_negotiation.cc:407] Connection header received
0812 16:38:10.560008 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:10.560012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:10.560109 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:10.560205 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:10.560611 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.561426 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.562679 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.563180 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.563293 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.563849 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:10.563971 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:10.564055 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:10.564200 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 16:38:10.564282 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":339,"threads_started":1}
I20260812 16:38:10.569944 834 raft_consensus.cc:1261] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Refusing update from remote peer 608afbc6433d4c9b93b0cb85484f6afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:10.570035 1119 raft_consensus.cc:1261] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 1 FOLLOWER]: Refusing update from remote peer 608afbc6433d4c9b93b0cb85484f6afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:10.570660 1203 consensus_queue.cc:1035] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:10.571103 1192 consensus_queue.cc:1035] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:10.578094 1191 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: TxnStatusTablet state changed. Reason: New leader 608afbc6433d4c9b93b0cb85484f6afa. Latest consensus state: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.578416 1191 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:10.579360 1192 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.579735 1192 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:38:10.586021 1191 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.586308 1191 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P a3cd30cbbb7e4644b5b5749aa48b8ce6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:10.592499 1190 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: TxnStatusTablet state changed. Reason: New leader 608afbc6433d4c9b93b0cb85484f6afa. Latest consensus state: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.592803 1190 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:10.594558 1203 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.594848 1203 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P 608afbc6433d4c9b93b0cb85484f6afa: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:38:10.598997 1190 tablet_replica.cc:439] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "608afbc6433d4c9b93b0cb85484f6afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 } } peers { permanent_uuid: "608afbc6433d4c9b93b0cb85484f6afa" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33181 } } peers { permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 } } }
I20260812 16:38:10.599275 1190 tablet_replica.cc:442] T dc8d12c6aa604a52825bad71508df775 P b8824b26e9b6480f9d326c2fd4a7450b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:10.611327 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.604608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:43433 (local address 127.31.61.130:33181)
0812 16:38:10.604799 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:10.604807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:10.605345 (+ 538us) server_negotiation.cc:407] Connection header received
0812 16:38:10.605556 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:10.605559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:10.605620 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:10.605689 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:10.606359 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.607145 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.609282 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.609732 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.609836 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.610457 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:10.610601 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:10.610672 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:10.610994 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 16:38:10.611076 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 16:38:10.611480 1239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.604506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:33181 (local address 127.31.61.130:43433)
0812 16:38:10.605244 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:10.605263 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:10.605415 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:10.605718 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:10.605723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:10.605738 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:38:10.606159 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:10.606171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.607268 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.607274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:10.609127 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:10.609144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.609913 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.609920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:10.610110 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.610187 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:38:10.610701 (+ 514us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:38:10.610719 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 16:38:10.610828 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 16:38:10.610988 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":384,"threads_started":1}
I20260812 16:38:10.618198 1192 consensus_queue.cc:1035] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cd30cbbb7e4644b5b5749aa48b8ce6" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:10.625442 832 raft_consensus.cc:1261] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Refusing update from remote peer 608afbc6433d4c9b93b0cb85484f6afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:10.626119 1240 consensus_queue.cc:1035] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8824b26e9b6480f9d326c2fd4a7450b" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:10.696352 672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:10.696646 672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:10.700040 672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 616192fc9c0c43469aaa0e52e4575b6f: Sending DeleteTablet for 3 replicas of tablet 9f795d0adfb24fb9976a6b8e5d81d892
I20260812 16:38:10.700937 810 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f795d0adfb24fb9976a6b8e5d81d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:10 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:41536
I20260812 16:38:10.700991 952 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f795d0adfb24fb9976a6b8e5d81d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:10 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:49298
I20260812 16:38:10.702158 1093 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f795d0adfb24fb9976a6b8e5d81d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:10 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:60184
I20260812 16:38:10.710618 1248 tablet_replica.cc:330] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: stopping tablet replica
I20260812 16:38:10.711061 1248 raft_consensus.cc:2229] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:10.711696 1248 raft_consensus.cc:2258] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:10.713424 1248 ts_tablet_manager.cc:1905] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:10.716012 1246 tablet_replica.cc:330] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: stopping tablet replica
I20260812 16:38:10.718508 1246 raft_consensus.cc:2229] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:10.718910 1246 raft_consensus.cc:2258] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:10.720646 1246 ts_tablet_manager.cc:1905] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:10.723531 1248 ts_tablet_manager.cc:1918] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:38:10.723773 1248 log.cc:1196] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/9f795d0adfb24fb9976a6b8e5d81d892
I20260812 16:38:10.724291 1248 ts_tablet_manager.cc:1939] T 9f795d0adfb24fb9976a6b8e5d81d892 P a3cd30cbbb7e4644b5b5749aa48b8ce6: Deleting consensus metadata
I20260812 16:38:10.725567 656 catalog_manager.cc:4841] TS a3cd30cbbb7e4644b5b5749aa48b8ce6 (127.31.61.131:36177): tablet 9f795d0adfb24fb9976a6b8e5d81d892 (table test-table [id=54dcd5c1f5e749f9a2043c8f4766ad3d]) successfully deleted
I20260812 16:38:10.726588 1247 tablet_replica.cc:330] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: stopping tablet replica
I20260812 16:38:10.726943 1247 raft_consensus.cc:2229] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:10.727763 1247 raft_consensus.cc:2258] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:10.729017 1247 ts_tablet_manager.cc:1905] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:10.736792 1247 ts_tablet_manager.cc:1918] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:38:10.737154 1247 log.cc:1196] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/9f795d0adfb24fb9976a6b8e5d81d892
I20260812 16:38:10.737941 1247 ts_tablet_manager.cc:1939] T 9f795d0adfb24fb9976a6b8e5d81d892 P 608afbc6433d4c9b93b0cb85484f6afa: Deleting consensus metadata
I20260812 16:38:10.739457 658 catalog_manager.cc:4841] TS 608afbc6433d4c9b93b0cb85484f6afa (127.31.61.130:33181): tablet 9f795d0adfb24fb9976a6b8e5d81d892 (table test-table [id=54dcd5c1f5e749f9a2043c8f4766ad3d]) successfully deleted
I20260812 16:38:10.742197 1246 ts_tablet_manager.cc:1918] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:38:10.742501 1246 log.cc:1196] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/9f795d0adfb24fb9976a6b8e5d81d892
I20260812 16:38:10.743026 1246 ts_tablet_manager.cc:1939] T 9f795d0adfb24fb9976a6b8e5d81d892 P b8824b26e9b6480f9d326c2fd4a7450b: Deleting consensus metadata
I20260812 16:38:10.744208 657 catalog_manager.cc:4841] TS b8824b26e9b6480f9d326c2fd4a7450b (127.31.61.129:41603): tablet 9f795d0adfb24fb9976a6b8e5d81d892 (table test-table [id=54dcd5c1f5e749f9a2043c8f4766ad3d]) successfully deleted
Aug 12 16:38:10 dist-test-slave-f7th krb5kdc[625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:10.754531 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.715625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.31.61.129:41603)
0812 16:38:10.734636 (+ 19011us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:10.734643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:10.734669 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:38:10.734733 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:10.734738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:10.734820 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:10.737114 (+ 2294us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:10.737314 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.738382 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.740614 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.741341 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.741457 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.746750 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:10.746772 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:10.746795 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:10.746849 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:10.751187 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.752115 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.752126 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.752130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.752306 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.752721 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.752726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.752730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.753227 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:10.753543 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:10.753836 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 16:38:10.754120 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18826,"thread_start_us":18684,"threads_started":1}
W20260812 16:38:10.755476 877 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41540
I20260812 16:38:10.772185 1254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.759019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.31.61.190:45657)
0812 16:38:10.759829 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:10.759834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:10.759849 (+ 15us) server_negotiation.cc:407] Connection header received
0812 16:38:10.759905 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:10.759909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:10.759979 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:10.760066 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:10.761352 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.762303 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.763771 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.764421 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.764573 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.766899 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:10.766905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:10.766908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:10.766945 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:10.769573 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.770226 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.770231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.770233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.770361 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.770728 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.770733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.770735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.771028 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:10.771267 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:10.771581 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 16:38:10.771842 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":491,"threads_started":1}
W20260812 16:38:10.772804 672 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42538
I20260812 16:38:10.774688 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 743
I20260812 16:38:10.790414 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.775958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.31.61.129:41603)
0812 16:38:10.776175 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:10.776179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:10.776297 (+ 118us) server_negotiation.cc:407] Connection header received
0812 16:38:10.776439 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:10.776445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:10.776512 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:10.776616 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:10.778016 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.778727 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.780487 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.780911 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.781019 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.783942 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:10.783948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:10.783952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:10.783994 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:10.787172 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.788152 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.788157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.788159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.788303 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.788820 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.788825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.788827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.789115 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:10.789315 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:10.789975 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 16:38:10.790167 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 16:38:10.791397 877 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41542
W20260812 16:38:10.792213 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 743: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:38:10.792436 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 743
I20260812 16:38:10.809484 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 886
Aug 12 16:38:10 dist-test-slave-f7th krb5kdc[625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:38:10.827706 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.810368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.31.61.130:33181)
0812 16:38:10.810569 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:10.810575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:10.810598 (+ 23us) server_negotiation.cc:407] Connection header received
0812 16:38:10.810795 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:10.810801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:10.810893 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:10.811021 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:10.812389 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.813597 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.815022 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.815792 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.815928 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.820199 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:10.820215 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:10.820233 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:10.820275 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:10.824128 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.825201 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.825254 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.825257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.825441 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.825992 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.825996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.825998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.826451 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:10.826724 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:10.827181 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 16:38:10.827363 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260812 16:38:10.828459 1019 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49320
W20260812 16:38:10.829144 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 886: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:38:10.829299 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 886
I20260812 16:38:10.852360 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1028
Aug 12 16:38:10 dist-test-slave-f7th krb5kdc[625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:38:10.869199 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:10.853175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.31.61.131:36177)
0812 16:38:10.853754 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:10.853760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:10.853778 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:38:10.853836 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:10.853840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:10.853909 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:10.854033 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:10.855638 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.856448 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.857704 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:10.858387 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:10.858593 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:10.862955 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:10.862971 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:10.862990 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:10.863040 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:10.866203 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.866977 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.866985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.866988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.867173 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:10.867544 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:10.867548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:10.867550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:10.868050 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:10.868321 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:10.868629 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 16:38:10.868814 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":335,"threads_started":1}
W20260812 16:38:10.869859 1159 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60194
W20260812 16:38:10.870597 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1028: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:38:10.870743 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 1028
I20260812 16:38:10.886502 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 638
I20260812 16:38:10.888051 712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:11.006438 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 638
2026-08-12T16:38:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4988 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:11 dist-test-slave-f7th krb5kdc[1260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:11 dist-test-slave-f7th krb5kdc[1260](info): set up 2 sockets
Aug 12 16:38:11 dist-test-slave-f7th krb5kdc[1260](info): commencing operation
krb5kdc: starting...
W20260812 16:38:12.027426 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:12 dist-test-slave-f7th krb5kdc[1260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
W20260812 16:38:13.019795 31990 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.61.190: real 0.829s user 0.003s sys 0.008s
I20260812 16:38:13.020869 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:45007
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:13.463088 1273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:13.463413 1273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:13.463550 1273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:13.471503 1273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:13.471668 1273 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 16:38:13.471726 1273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:13.471813 1273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:13.471869 1273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:13.471935 1273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:13.482206 1273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:45007
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:13.483589 1273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:13.485029 1273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:13.497568 1279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:13.497572 1278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:13.498366 1282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:13.498461 1273 server_base.cc:1034] running on GCE node
I20260812 16:38:13.499346 1273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:13.501128 1273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:13.502418 1273 hybrid_clock.cc:648] HybridClock initialized: now 1786552693502352 us; error 34 us; skew 500 ppm
I20260812 16:38:13.502830 1273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:13 dist-test-slave-f7th krb5kdc[1260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552693, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:13.511415 1273 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:38:13.513744 1273 webserver.cc:460] Webserver started at http://127.31.61.190:37183/ using document root <none> and password file <none>
I20260812 16:38:13.514698 1273 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:13.514824 1273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:13.515175 1273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:13.518402 1273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "07263668d7c14bf592cfeef03f51ab4e"
format_stamp: "Formatted at 2026-08-12 16:38:13 on dist-test-slave-f7th"
server_key: "da6ca279a440c1f41b4283d4aad8e306"
server_key_iv: "06782a8fa2b158ff83c69d476f0ba94c"
server_key_version: "encryptionkey@0"
I20260812 16:38:13.519202 1273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "07263668d7c14bf592cfeef03f51ab4e"
format_stamp: "Formatted at 2026-08-12 16:38:13 on dist-test-slave-f7th"
server_key: "da6ca279a440c1f41b4283d4aad8e306"
server_key_iv: "06782a8fa2b158ff83c69d476f0ba94c"
server_key_version: "encryptionkey@0"
I20260812 16:38:13.524930 1273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 16:38:13.529348 1289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:13.531332 1273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:38:13.531505 1273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "07263668d7c14bf592cfeef03f51ab4e"
format_stamp: "Formatted at 2026-08-12 16:38:13 on dist-test-slave-f7th"
server_key: "da6ca279a440c1f41b4283d4aad8e306"
server_key_iv: "06782a8fa2b158ff83c69d476f0ba94c"
server_key_version: "encryptionkey@0"
I20260812 16:38:13.531816 1273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:13.575692 1273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:13.579931 1273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:13.580159 1273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:13.601538 1273 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:45007
I20260812 16:38:13.601584 1350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:45007 every 8 connection(s)
I20260812 16:38:13.602885 1273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:13.605860 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 1273
I20260812 16:38:13.606094 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:13.606866 31990 external_mini_cluster.cc:1428] Setting key f04688538e6aebde3168a9fe80f2c92c
I20260812 16:38:13.607067 1351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 16:38:13 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:13.622408 1351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e: Bootstrap starting.
I20260812 16:38:13.626116 1351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:13.627214 1351 log.cc:824] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:13.630170 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:13.608950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.31.61.190:45007)
0812 16:38:13.609739 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:13.609751 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:13.609790 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:38:13.612809 (+ 3019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:13.612833 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:13.613227 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:13.613600 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:13.615071 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:13.616388 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:13.617665 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:13.618375 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:13.618547 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:13.622890 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:13.622899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:13.622918 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:13.622960 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:13.626000 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:13.626694 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:13.626701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:13.626707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:13.626872 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:13.627242 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:13.627246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:13.627248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:13.627885 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:13.628233 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:13.628737 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 16:38:13.629163 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":377,"threads_started":1}
I20260812 16:38:13.630273 1351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e: No bootstrap required, opened a new log
I20260812 16:38:13.639670 1351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07263668d7c14bf592cfeef03f51ab4e" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45007 } }
I20260812 16:38:13.640089 1351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:13.640183 1351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07263668d7c14bf592cfeef03f51ab4e, State: Initialized, Role: FOLLOWER
I20260812 16:38:13.640854 1351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07263668d7c14bf592cfeef03f51ab4e" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45007 } }
I20260812 16:38:13.641206 1351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:13.641343 1351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:13.641494 1351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:13.711879 1351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07263668d7c14bf592cfeef03f51ab4e" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45007 } }
I20260812 16:38:13.712704 1351 leader_election.cc:302] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07263668d7c14bf592cfeef03f51ab4e; no voters:
I20260812 16:38:13.713479 1351 leader_election.cc:288] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:13.713584 1356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:13.716490 1356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [term 1 LEADER]: Becoming Leader. State: Replica: 07263668d7c14bf592cfeef03f51ab4e, State: Running, Role: LEADER
I20260812 16:38:13.717056 1356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07263668d7c14bf592cfeef03f51ab4e" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45007 } }
I20260812 16:38:13.717510 1351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:13.719827 1358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07263668d7c14bf592cfeef03f51ab4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07263668d7c14bf592cfeef03f51ab4e" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45007 } } }
I20260812 16:38:13.719805 1357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07263668d7c14bf592cfeef03f51ab4e. Latest consensus state: current_term: 1 leader_uuid: "07263668d7c14bf592cfeef03f51ab4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07263668d7c14bf592cfeef03f51ab4e" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45007 } } }
I20260812 16:38:13.720333 1358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:13.720397 1357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:13.722307 1362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:13.728060 1362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:13.738801 1362 catalog_manager.cc:1347] Generated new cluster ID: d275ec0e8be847c0ae40888537d68277
I20260812 16:38:13.738935 1362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:13.749281 1362 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:13.750810 1362 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:13.761389 1362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e: Generated new TSK 0
I20260812 16:38:13.762292 1362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
W20260812 16:38:14.071625 31990 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.61.129: real 0.296s user 0.004s sys 0.006s
I20260812 16:38:14.072762 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45007
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:14.433125 1378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:14.433408 1378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:14.433540 1378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:14.441300 1378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:14.441425 1378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:14.441540 1378 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 16:38:14.441609 1378 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:38:14.441715 1378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:14.452203 1378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.61.190:45007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:14.453603 1378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:14.455013 1378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:14.469018 1383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:14.469022 1384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:14.469208 1386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:14.469990 1378 server_base.cc:1034] running on GCE node
I20260812 16:38:14.470669 1378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:14.471813 1378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:14.473069 1378 hybrid_clock.cc:648] HybridClock initialized: now 1786552694473012 us; error 47 us; skew 500 ppm
I20260812 16:38:14.473430 1378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:14 dist-test-slave-f7th krb5kdc[1260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552694, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:14.476689 1378 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:38:14.478223 1378 webserver.cc:460] Webserver started at http://127.31.61.129:35087/ using document root <none> and password file <none>
I20260812 16:38:14.478891 1378 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:14.478996 1378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:14.479264 1378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:14.482359 1378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "576018e1f4e84fab94f555491f3cbfe7"
format_stamp: "Formatted at 2026-08-12 16:38:14 on dist-test-slave-f7th"
server_key: "1ed6ecaeb667aa54c97acc8021ee515a"
server_key_iv: "531200315ae2cb1c9fa7ce17178b38da"
server_key_version: "encryptionkey@0"
I20260812 16:38:14.483096 1378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "576018e1f4e84fab94f555491f3cbfe7"
format_stamp: "Formatted at 2026-08-12 16:38:14 on dist-test-slave-f7th"
server_key: "1ed6ecaeb667aa54c97acc8021ee515a"
server_key_iv: "531200315ae2cb1c9fa7ce17178b38da"
server_key_version: "encryptionkey@0"
I20260812 16:38:14.488817 1378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 16:38:14.492640 1393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:14.494045 1378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:14.494213 1378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "576018e1f4e84fab94f555491f3cbfe7"
format_stamp: "Formatted at 2026-08-12 16:38:14 on dist-test-slave-f7th"
server_key: "1ed6ecaeb667aa54c97acc8021ee515a"
server_key_iv: "531200315ae2cb1c9fa7ce17178b38da"
server_key_version: "encryptionkey@0"
I20260812 16:38:14.494422 1378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:14.505756 1378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:14.509994 1378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:14.510275 1378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:14.513137 1378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:14.513245 1378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:14.513417 1378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:14.513497 1378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:38:14 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552694, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:14.538759 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:14.513522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:55683 (local address 127.31.61.190:45007)
0812 16:38:14.514033 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:14.514040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:14.515309 (+ 1269us) server_negotiation.cc:407] Connection header received
0812 16:38:14.517448 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:14.517453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:14.517557 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:14.517686 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:14.519877 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:14.520773 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:14.523376 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:14.523957 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:14.524085 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:14.529369 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:14.529379 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:14.529384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:14.529435 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:14.532957 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:14.534102 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:14.534106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:14.534109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:14.534246 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:14.534880 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:14.534887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:14.534891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:14.535322 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:14.535478 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:14.537938 (+ 2460us) server_negotiation.cc:299] Negotiation successful
0812 16:38:14.538227 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
I20260812 16:38:14.539165 1404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:14.514209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45007 (local address 127.31.61.129:55683)
0812 16:38:14.515109 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:14.515163 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:14.517167 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:14.517958 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:14.517968 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:14.518739 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:14.519682 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:14.519713 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:14.520935 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:14.520941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:14.523214 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:14.523231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:14.524323 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:14.524336 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:14.524566 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:14.525351 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:14.525396 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:14.529041 (+ 3645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:14.533177 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:14.533189 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:14.533217 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:14.533937 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:14.534376 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:14.534387 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:14.534392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:14.534732 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:14.535588 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:14.535628 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:14.536260 (+ 632us) client_negotiation.cc:769] Sending connection context
0812 16:38:14.536666 (+ 406us) client_negotiation.cc:240] Negotiation successful
0812 16:38:14.537346 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":418,"threads_started":1}
I20260812 16:38:14.555140 1378 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:45687
I20260812 16:38:14.555235 1515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:45687 every 8 connection(s)
I20260812 16:38:14.556890 1378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:14.559171 1516 heartbeater.cc:347] Connected to a master server at 127.31.61.190:45007
I20260812 16:38:14.559445 1516 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:14.560165 1516 heartbeater.cc:511] Master 127.31.61.190:45007 requested a full tablet report, sending...
I20260812 16:38:14.562635 1306 ts_manager.cc:194] Registered new tserver with Master: 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687)
I20260812 16:38:14.562916 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:14.547096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45007 (local address 127.31.61.129:39007)
0812 16:38:14.548125 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:14.548149 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:14.548315 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:14.548680 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:14.548686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:14.549492 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:14.549982 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:14.550000 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:14.551562 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:14.551569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:14.553352 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:14.553368 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:14.554468 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:14.554476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:14.554685 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:14.555470 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:14.555496 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:14.556569 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:14.560252 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:14.560258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:14.560262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:14.560676 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:14.561088 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:14.561103 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:14.561107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:14.561277 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:14.561992 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:14.561997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:14.562161 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 16:38:14.562282 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 16:38:14.562563 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":682,"threads_started":1}
I20260812 16:38:14.563016 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:14.547122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:39007 (local address 127.31.61.190:45007)
0812 16:38:14.547360 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:14.547368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:14.548254 (+ 886us) server_negotiation.cc:407] Connection header received
0812 16:38:14.548442 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:14.548446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:14.548516 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:14.548623 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:14.550219 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:14.551359 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:14.553506 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:14.554141 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:14.554297 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:14.556789 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:14.556794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:14.556798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:14.556834 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:14.560122 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:14.560801 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:14.560807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:14.560811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:14.560963 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:14.561444 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:14.561449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:14.561452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:14.561847 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:14.562029 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:14.562432 (+ 403us) server_negotiation.cc:299] Negotiation successful
0812 16:38:14.562675 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:38:14.564100 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 1378
I20260812 16:38:14.564301 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:14.564927 31990 external_mini_cluster.cc:1428] Setting key 34fcc6849c4d807ee350e6aa0bc47b70
I20260812 16:38:14.566519 1306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:55683
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:38:14.611306 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45007
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:14.942713 1520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:14.943027 1520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:14.943146 1520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:14.950531 1520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:14.950659 1520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:14.950748 1520 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 16:38:14.950799 1520 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:38:14.950868 1520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:14.960938 1520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.61.190:45007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:14.962265 1520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:14.963567 1520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:14.977988 1525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:14.978018 1526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:14.978039 1528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:14.979084 1520 server_base.cc:1034] running on GCE node
I20260812 16:38:14.979797 1520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:14.980759 1520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:14.981986 1520 hybrid_clock.cc:648] HybridClock initialized: now 1786552694981935 us; error 40 us; skew 500 ppm
I20260812 16:38:14.982321 1520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:14 dist-test-slave-f7th krb5kdc[1260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552694, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:14.985392 1520 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:38:14.986867 1520 webserver.cc:460] Webserver started at http://127.31.61.130:33055/ using document root <none> and password file <none>
I20260812 16:38:14.987545 1520 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:14.987686 1520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:14.987982 1520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:14.990826 1520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "8ed430d674974765b2c4ddf84cb77a5c"
format_stamp: "Formatted at 2026-08-12 16:38:14 on dist-test-slave-f7th"
server_key: "7eb66d34d54e5eae1828705fde324488"
server_key_iv: "0a9316e4f4aea7dc0963fdb4b621f246"
server_key_version: "encryptionkey@0"
I20260812 16:38:14.991551 1520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "8ed430d674974765b2c4ddf84cb77a5c"
format_stamp: "Formatted at 2026-08-12 16:38:14 on dist-test-slave-f7th"
server_key: "7eb66d34d54e5eae1828705fde324488"
server_key_iv: "0a9316e4f4aea7dc0963fdb4b621f246"
server_key_version: "encryptionkey@0"
I20260812 16:38:14.996924 1520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 16:38:15.001180 1535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:15.002715 1520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:15.002895 1520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "8ed430d674974765b2c4ddf84cb77a5c"
format_stamp: "Formatted at 2026-08-12 16:38:14 on dist-test-slave-f7th"
server_key: "7eb66d34d54e5eae1828705fde324488"
server_key_iv: "0a9316e4f4aea7dc0963fdb4b621f246"
server_key_version: "encryptionkey@0"
I20260812 16:38:15.003106 1520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:15.040279 1520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:15.045814 1520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:15.046151 1520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:15.049224 1520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:15.049345 1520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:15.049470 1520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:15.049556 1520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:38:15 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552694, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:15.074870 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.050229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45007 (local address 127.31.61.130:35067)
0812 16:38:15.051111 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.051167 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.053100 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.053992 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.054006 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.054891 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:15.055885 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.055912 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.057284 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.057292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.059348 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.059366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.060509 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.060518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.060858 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.061814 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:15.061854 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:15.064969 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:15.069045 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.069052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.069067 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.069705 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.070174 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.070179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.070183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.070499 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.071286 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:15.071294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:15.071879 (+ 585us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.072319 (+ 440us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.073105 (+ 786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":399,"threads_started":1}
I20260812 16:38:15.076294 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.049614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:35067 (local address 127.31.61.190:45007)
0812 16:38:15.049822 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.049830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.051322 (+ 1492us) server_negotiation.cc:407] Connection header received
0812 16:38:15.053490 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.053496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.053596 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.053715 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:15.056093 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.057072 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.059517 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.060221 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.060368 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.065296 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:15.065303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:15.065308 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:15.065348 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:15.068856 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.069888 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.069894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.069898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.070057 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.070665 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.070671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.070674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.071034 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:15.071208 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.075532 (+ 4324us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.075856 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 16:38:15.102023 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.086119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45007 (local address 127.31.61.130:59935)
0812 16:38:15.086839 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.086862 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.087039 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.087359 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.087364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.088024 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:15.088424 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.088439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.089489 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.089494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.091409 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.091429 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.092477 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.092484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.092663 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.093481 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:15.093509 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:15.094868 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:15.098772 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.098777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.098780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.099256 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.099941 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.099953 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.099957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.100154 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.100859 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:15.100863 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:15.101131 (+ 268us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.101342 (+ 211us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.101684 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":360,"threads_started":1}
I20260812 16:38:15.102133 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.086077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:59935 (local address 127.31.61.190:45007)
0812 16:38:15.086300 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.086308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.086995 (+ 687us) server_negotiation.cc:407] Connection header received
0812 16:38:15.087153 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.087157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.087226 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.087313 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:15.088616 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.089361 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.091589 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.092229 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.092393 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.095119 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:15.095126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:15.095130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:15.095182 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:15.098627 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.099581 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.099587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.099590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.099799 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.100277 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.100283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.100287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.100740 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:15.100901 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.101491 (+ 590us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.101765 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:38:15.104940 1520 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:43197
I20260812 16:38:15.105000 1657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:43197 every 8 connection(s)
I20260812 16:38:15.106297 1520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:15.108800 1658 heartbeater.cc:347] Connected to a master server at 127.31.61.190:45007
I20260812 16:38:15.109053 1658 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:15.109740 1658 heartbeater.cc:511] Master 127.31.61.190:45007 requested a full tablet report, sending...
I20260812 16:38:15.111197 1306 ts_manager.cc:194] Registered new tserver with Master: 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197)
I20260812 16:38:15.112099 1306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:35067
I20260812 16:38:15.112744 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 1520
I20260812 16:38:15.112895 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:15.113304 31990 external_mini_cluster.cc:1428] Setting key 549c471eff64748432025a75f4186ea2
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:38:15.160549 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45007
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:15.495638 1662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:15.495934 1662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:15.496048 1662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:15.503868 1662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:15.504025 1662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:15.504122 1662 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 16:38:15.504182 1662 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:38:15.504257 1662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:15.514813 1662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.61.190:45007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:15.516178 1662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:15.517586 1662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:15.531183 1667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:15.531553 1668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:15.531584 1670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:15.532684 1662 server_base.cc:1034] running on GCE node
I20260812 16:38:15.533321 1662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:15.534293 1662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:15.535511 1662 hybrid_clock.cc:648] HybridClock initialized: now 1786552695535460 us; error 43 us; skew 500 ppm
I20260812 16:38:15.535880 1662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:15 dist-test-slave-f7th krb5kdc[1260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552695, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:15.539069 1662 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:38:15.540657 1662 webserver.cc:460] Webserver started at http://127.31.61.131:35365/ using document root <none> and password file <none>
I20260812 16:38:15.541385 1662 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:15.541507 1662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:15.541812 1662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:15.545308 1662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "e07fa288f8844b2d8236febe291ab53d"
format_stamp: "Formatted at 2026-08-12 16:38:15 on dist-test-slave-f7th"
server_key: "dd2be0c326ffe3ae8e498b412df22aae"
server_key_iv: "6344733e333ee5f37c2686843df5d9d7"
server_key_version: "encryptionkey@0"
I20260812 16:38:15.546286 1662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "e07fa288f8844b2d8236febe291ab53d"
format_stamp: "Formatted at 2026-08-12 16:38:15 on dist-test-slave-f7th"
server_key: "dd2be0c326ffe3ae8e498b412df22aae"
server_key_iv: "6344733e333ee5f37c2686843df5d9d7"
server_key_version: "encryptionkey@0"
I20260812 16:38:15.552109 1662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 16:38:15.555847 1677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:15.557441 1662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:15.557658 1662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "e07fa288f8844b2d8236febe291ab53d"
format_stamp: "Formatted at 2026-08-12 16:38:15 on dist-test-slave-f7th"
server_key: "dd2be0c326ffe3ae8e498b412df22aae"
server_key_iv: "6344733e333ee5f37c2686843df5d9d7"
server_key_version: "encryptionkey@0"
I20260812 16:38:15.557845 1662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:15.569447 1516 heartbeater.cc:503] Master 127.31.61.190:45007 was elected leader, sending a full tablet report...
I20260812 16:38:15.598237 1662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:15.602856 1662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:15.603135 1662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:15.606184 1662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:15.606295 1662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:15.606410 1662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:15.606488 1662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:38:15 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552695, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:15.633600 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.607085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:38197 (local address 127.31.61.190:45007)
0812 16:38:15.612055 (+ 4970us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.612062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.612089 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:38:15.613731 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.613737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.613828 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.613938 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:15.616374 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.617433 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.620162 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.620752 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.620898 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.626317 (+ 5419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:15.626325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:15.626331 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:15.626393 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:15.629900 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.630805 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.630813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.630817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.630979 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.631490 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.631497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.631501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.631883 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:15.632045 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.632870 (+ 825us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.633127 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4822,"thread_start_us":4714,"threads_started":1}
I20260812 16:38:15.634642 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.607752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45007 (local address 127.31.61.131:38197)
0812 16:38:15.611187 (+ 3435us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.611250 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.613432 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.614231 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.614244 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.615105 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:15.616092 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.616123 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.617625 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.617632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.619992 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.620006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.621058 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.621067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.621469 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.622342 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:15.622382 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:15.625978 (+ 3596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:15.630059 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.630070 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.630091 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.630661 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.631100 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.631106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.631111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.631356 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.632034 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:15.632044 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:15.632443 (+ 399us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.632719 (+ 276us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.633282 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3130,"thread_start_us":2906,"threads_started":1}
I20260812 16:38:15.662804 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.647556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:50911 (local address 127.31.61.190:45007)
0812 16:38:15.647817 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.647825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.648623 (+ 798us) server_negotiation.cc:407] Connection header received
0812 16:38:15.648858 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.648865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.648943 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.649036 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:15.650139 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.650938 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.652863 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.653512 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.653645 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.656279 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:15.656286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:15.656291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:15.656329 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:15.659592 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.660524 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.660529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.660532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.660701 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.661075 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.661079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.661082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.661475 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:15.661709 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.662175 (+ 466us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.662437 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 16:38:15.662827 1767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.647772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45007 (local address 127.31.61.131:50911)
0812 16:38:15.648478 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.648505 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.648694 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.649086 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.649091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.649608 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:15.649978 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.649990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.651093 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.651098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.652614 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.652645 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.653727 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.653735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.653938 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.654670 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:15.654700 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:15.656040 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:15.659809 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.659814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.659819 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.660360 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.660823 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.660827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.660829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.660954 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.661650 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:15.661655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:15.661892 (+ 237us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.662023 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.662324 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":343,"threads_started":1}
I20260812 16:38:15.664690 1662 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:38493
I20260812 16:38:15.664806 1798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:38493 every 8 connection(s)
I20260812 16:38:15.666904 1662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:15.669703 1799 heartbeater.cc:347] Connected to a master server at 127.31.61.190:45007
I20260812 16:38:15.669957 1799 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:15.670763 1799 heartbeater.cc:511] Master 127.31.61.190:45007 requested a full tablet report, sending...
I20260812 16:38:15.671622 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 1662
I20260812 16:38:15.671824 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:15.672375 1306 ts_manager.cc:194] Registered new tserver with Master: e07fa288f8844b2d8236febe291ab53d (127.31.61.131:38493)
I20260812 16:38:15.672436 31990 external_mini_cluster.cc:1428] Setting key f701cae90cd5c984a463a16b07d80084
I20260812 16:38:15.673398 1306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:38197
I20260812 16:38:15.675814 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:38:15 dist-test-slave-f7th krb5kdc[1260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:38:15 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:15.725100 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.711024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.31.61.190:45007)
0812 16:38:15.711275 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.711281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.711587 (+ 306us) server_negotiation.cc:407] Connection header received
0812 16:38:15.711828 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.711832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.711899 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.711985 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:15.713393 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.714116 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.715310 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.715761 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.715875 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.719923 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:15.719928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:15.719932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:15.719970 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:15.722455 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.723137 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.723142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.723144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.723277 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.723677 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.723683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.723686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.723977 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:15.724255 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.724527 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.724760 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 16:38:15.729761 1306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:15.737392 1306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:15.774521 1815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.752066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:45687 (local address 127.0.0.1:35978)
0812 16:38:15.764153 (+ 12087us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.764217 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.764464 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.766063 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.766072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.766118 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:15.767265 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.767279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.768745 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.768751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.771166 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.771183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.773437 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.773450 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.773698 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.773754 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.773913 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.774122 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10312,"thread_start_us":13847,"threads_started":1}
I20260812 16:38:15.775753 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.752612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.31.61.129:45687)
0812 16:38:15.765343 (+ 12731us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.765352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.765387 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:38:15.765510 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.765520 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.765814 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.766023 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:15.767474 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.768597 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.771455 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.773128 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.773270 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.773493 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.775213 (+ 1720us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.775415 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12551,"thread_start_us":12409,"threads_started":1}
I20260812 16:38:15.775765 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.752569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:38493 (local address 127.0.0.1:52540)
0812 16:38:15.766023 (+ 13454us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.766186 (+ 163us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.766442 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.767586 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.767592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.767646 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:15.768310 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.768328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.770255 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.770260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.772772 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.772788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.774980 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.774986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.775190 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.775223 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.775297 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.775434 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13226,"mutex_wait_us":35,"thread_start_us":12244,"threads_started":1}
I20260812 16:38:15.775799 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.753738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.31.61.130:43197)
0812 16:38:15.763924 (+ 10186us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.763936 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.766556 (+ 2620us) server_negotiation.cc:407] Connection header received
0812 16:38:15.767018 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.767027 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.767233 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.767434 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:15.768553 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.769979 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.772675 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.774411 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.774548 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.774745 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.775313 (+ 568us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.775471 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9996,"thread_start_us":9858,"threads_started":1}
I20260812 16:38:15.776495 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.753235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.31.61.131:38493)
0812 16:38:15.765866 (+ 12631us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.765874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.766288 (+ 414us) server_negotiation.cc:407] Connection header received
0812 16:38:15.766767 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.766776 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.767014 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.767222 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:15.768571 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.770081 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.772985 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.774478 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.774611 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.774830 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.776060 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.776238 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12445,"thread_start_us":12236,"threads_started":1}
I20260812 16:38:15.776525 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.753008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:43197 (local address 127.0.0.1:47830)
0812 16:38:15.766411 (+ 13403us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.766437 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.766657 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.767654 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.767659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.767679 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:15.768332 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.768342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.770179 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.770185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.772498 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.772513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.774874 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.774880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.775091 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.775123 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.776048 (+ 925us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.776200 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13118,"mutex_wait_us":124,"thread_start_us":9637,"threads_started":1}
I20260812 16:38:15.779639 1586 tablet_service.cc:1467] Processing CreateTablet for tablet 585e8694e5d541d2ab3f32272b919fa0 (DEFAULT_TABLE table=test-table [id=0a87310b8b0a462e953ee6aae1dc0b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:15.779506 1445 tablet_service.cc:1467] Processing CreateTablet for tablet 585e8694e5d541d2ab3f32272b919fa0 (DEFAULT_TABLE table=test-table [id=0a87310b8b0a462e953ee6aae1dc0b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:15.780817 1445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585e8694e5d541d2ab3f32272b919fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:15.780845 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585e8694e5d541d2ab3f32272b919fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:15.786605 1729 tablet_service.cc:1467] Processing CreateTablet for tablet 585e8694e5d541d2ab3f32272b919fa0 (DEFAULT_TABLE table=test-table [id=0a87310b8b0a462e953ee6aae1dc0b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:15.788364 1729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585e8694e5d541d2ab3f32272b919fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:15.794461 1818 tablet_bootstrap.cc:492] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: Bootstrap starting.
I20260812 16:38:15.795238 1819 tablet_bootstrap.cc:492] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: Bootstrap starting.
I20260812 16:38:15.798545 1818 tablet_bootstrap.cc:654] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:15.799662 1819 tablet_bootstrap.cc:654] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:15.800275 1818 log.cc:824] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:15.800940 1819 log.cc:824] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:15.803081 1822 tablet_bootstrap.cc:492] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: Bootstrap starting.
I20260812 16:38:15.807070 1822 tablet_bootstrap.cc:654] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:15.807236 1819 tablet_bootstrap.cc:492] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: No bootstrap required, opened a new log
I20260812 16:38:15.807659 1819 ts_tablet_manager.cc:1397] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260812 16:38:15.807998 1818 tablet_bootstrap.cc:492] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: No bootstrap required, opened a new log
I20260812 16:38:15.808303 1818 ts_tablet_manager.cc:1397] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260812 16:38:15.808888 1822 log.cc:824] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:15.818246 1822 tablet_bootstrap.cc:492] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: No bootstrap required, opened a new log
I20260812 16:38:15.818558 1822 ts_tablet_manager.cc:1397] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260812 16:38:15.819269 1819 raft_consensus.cc:348] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } }
I20260812 16:38:15.819270 1818 raft_consensus.cc:348] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } }
I20260812 16:38:15.819787 1819 raft_consensus.cc:374] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:15.819789 1818 raft_consensus.cc:374] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:15.819895 1819 raft_consensus.cc:729] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576018e1f4e84fab94f555491f3cbfe7, State: Initialized, Role: FOLLOWER
I20260812 16:38:15.819895 1818 raft_consensus.cc:729] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed430d674974765b2c4ddf84cb77a5c, State: Initialized, Role: FOLLOWER
I20260812 16:38:15.820559 1818 consensus_queue.cc:260] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } }
I20260812 16:38:15.820547 1819 consensus_queue.cc:260] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } }
I20260812 16:38:15.824432 1658 heartbeater.cc:503] Master 127.31.61.190:45007 was elected leader, sending a full tablet report...
I20260812 16:38:15.825649 1818 ts_tablet_manager.cc:1428] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260812 16:38:15.826943 1819 ts_tablet_manager.cc:1428] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: Time spent starting tablet: real 0.019s user 0.007s sys 0.010s
I20260812 16:38:15.828815 1822 raft_consensus.cc:348] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } }
I20260812 16:38:15.829288 1822 raft_consensus.cc:374] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:15.829391 1822 raft_consensus.cc:729] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa288f8844b2d8236febe291ab53d, State: Initialized, Role: FOLLOWER
I20260812 16:38:15.830039 1822 consensus_queue.cc:260] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } }
I20260812 16:38:15.831365 1799 heartbeater.cc:503] Master 127.31.61.190:45007 was elected leader, sending a full tablet report...
I20260812 16:38:15.831923 1822 ts_tablet_manager.cc:1428] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260812 16:38:15.858701 1659 tablet.cc:2369] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:38:15.873538 1826 raft_consensus.cc:482] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:15.873847 1826 raft_consensus.cc:504] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } }
I20260812 16:38:15.875569 1826 leader_election.cc:288] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197)
I20260812 16:38:15.882545 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.876046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:53431 (local address 127.31.61.129:45687)
0812 16:38:15.876258 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.876265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.876519 (+ 254us) server_negotiation.cc:407] Connection header received
0812 16:38:15.876726 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.876731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.876820 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.876896 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:15.877325 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.878323 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.880678 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.881559 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.881716 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.881792 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.882144 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.882274 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:38:15.882597 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.876169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:44483 (local address 127.31.61.130:43197)
0812 16:38:15.876360 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.876365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.876972 (+ 607us) server_negotiation.cc:407] Connection header received
0812 16:38:15.877149 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.877153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.877209 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.877280 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:15.877711 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.878637 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.880926 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.881703 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.881849 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.881918 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.882221 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.882350 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:38:15.882717 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.876199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:43197 (local address 127.31.61.131:44483)
0812 16:38:15.876865 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.876880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.877012 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.877308 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.877313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.877332 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:15.877576 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.877583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.878754 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.878759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.880778 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.880788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.881912 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.881918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.882083 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.882107 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.882182 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.882315 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":28,"spinlock_wait_cycles":11520,"thread_start_us":355,"threads_started":1}
I20260812 16:38:15.882987 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.876096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:45687 (local address 127.31.61.131:53431)
0812 16:38:15.876411 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.876426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.876581 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.876947 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.876951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.877002 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:15.877193 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.877202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.878453 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.878457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.880523 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.880538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.881717 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.881723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.881955 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.882021 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.882104 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.882268 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":151,"spinlock_wait_cycles":22528}
I20260812 16:38:15.883782 1607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585e8694e5d541d2ab3f32272b919fa0" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed430d674974765b2c4ddf84cb77a5c" is_pre_election: true
I20260812 16:38:15.883909 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585e8694e5d541d2ab3f32272b919fa0" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7" is_pre_election: true
I20260812 16:38:15.884286 1607 raft_consensus.cc:2454] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 0.
I20260812 16:38:15.884348 1465 raft_consensus.cc:2454] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 0.
I20260812 16:38:15.884999 1681 leader_election.cc:302] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 576018e1f4e84fab94f555491f3cbfe7, 8ed430d674974765b2c4ddf84cb77a5c, e07fa288f8844b2d8236febe291ab53d; no voters:
I20260812 16:38:15.885407 1826 raft_consensus.cc:2781] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:15.885560 1826 raft_consensus.cc:482] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:15.885665 1826 raft_consensus.cc:3037] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:15.888078 1826 raft_consensus.cc:504] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } }
I20260812 16:38:15.888765 1826 leader_election.cc:288] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 election: Requested vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197)
I20260812 16:38:15.889263 1607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585e8694e5d541d2ab3f32272b919fa0" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed430d674974765b2c4ddf84cb77a5c"
I20260812 16:38:15.889276 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585e8694e5d541d2ab3f32272b919fa0" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7"
I20260812 16:38:15.889465 1607 raft_consensus.cc:3037] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:15.889472 1465 raft_consensus.cc:3037] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:15.891716 1607 raft_consensus.cc:2454] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 1.
I20260812 16:38:15.891788 1465 raft_consensus.cc:2454] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 1.
I20260812 16:38:15.892148 1681 leader_election.cc:302] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed430d674974765b2c4ddf84cb77a5c, e07fa288f8844b2d8236febe291ab53d; no voters:
I20260812 16:38:15.892463 1826 raft_consensus.cc:2781] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:15.892943 1826 raft_consensus.cc:686] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 1 LEADER]: Becoming Leader. State: Replica: e07fa288f8844b2d8236febe291ab53d, State: Running, Role: LEADER
I20260812 16:38:15.893559 1826 consensus_queue.cc:237] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } }
I20260812 16:38:15.898602 1305 catalog_manager.cc:5495] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d reported cstate change: term changed from 0 to 1, leader changed from <none> to e07fa288f8844b2d8236febe291ab53d (127.31.61.131). New cstate: current_term: 1 leader_uuid: "e07fa288f8844b2d8236febe291ab53d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } health_report { overall_health: UNKNOWN } } }
W20260812 16:38:15.919050 1800 tablet.cc:2369] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 16:38:15 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552693, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:15.973567 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.958294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.31.61.190:45007)
0812 16:38:15.958525 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:15.958531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:15.959063 (+ 532us) server_negotiation.cc:407] Connection header received
0812 16:38:15.959220 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:15.959224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:15.959286 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:15.959358 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:15.960586 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.961383 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.962674 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.963193 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.963296 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.967258 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:15.967265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:15.967270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:15.967419 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:15.970306 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.971282 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.971286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.971289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.971473 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:15.971980 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:15.971984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:15.971986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:15.972305 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:15.972467 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:15.973009 (+ 542us) server_negotiation.cc:299] Negotiation successful
0812 16:38:15.973200 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 16:38:15.973589 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:15.958309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45007 (local address 127.0.0.1:36942)
0812 16:38:15.958954 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:15.958972 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:15.959110 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:15.959391 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:15.959394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:15.960150 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:15.960416 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.960425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.961515 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.961519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.962494 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:15.962505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:15.963462 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:15.963472 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:15.963661 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:15.964379 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:15.964400 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:15.966880 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:15.970498 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.970588 (+ 90us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.970598 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.971143 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.971626 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:15.971633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:15.971637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:15.971842 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:15.972486 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:15.972496 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:15.972707 (+ 211us) client_negotiation.cc:769] Sending connection context
0812 16:38:15.972848 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 16:38:15.973092 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":376,"threads_started":1}
I20260812 16:38:15.979753 1305 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:36942:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 16:38:15.980859 1305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:15.988265 1586 tablet_service.cc:1467] Processing CreateTablet for tablet 7e2fd4fa5bf0433ba87aad214511950c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=483000d1999b4bf3be2c7319073b92cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:38:15.988358 1445 tablet_service.cc:1467] Processing CreateTablet for tablet 7e2fd4fa5bf0433ba87aad214511950c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=483000d1999b4bf3be2c7319073b92cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:38:15.988741 1729 tablet_service.cc:1467] Processing CreateTablet for tablet 7e2fd4fa5bf0433ba87aad214511950c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=483000d1999b4bf3be2c7319073b92cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:38:15.989317 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e2fd4fa5bf0433ba87aad214511950c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:15.989583 1729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e2fd4fa5bf0433ba87aad214511950c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:15.989569 1445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e2fd4fa5bf0433ba87aad214511950c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:15.995646 1818 tablet_bootstrap.cc:492] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: Bootstrap starting.
I20260812 16:38:15.997121 1818 tablet_bootstrap.cc:654] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:15.997619 1819 tablet_bootstrap.cc:492] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: Bootstrap starting.
I20260812 16:38:15.997749 1822 tablet_bootstrap.cc:492] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: Bootstrap starting.
I20260812 16:38:15.999374 1822 tablet_bootstrap.cc:654] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:15.999459 1819 tablet_bootstrap.cc:654] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:16.000098 1818 tablet_bootstrap.cc:492] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: No bootstrap required, opened a new log
I20260812 16:38:16.000250 1818 ts_tablet_manager.cc:1397] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 16:38:16.001142 1818 raft_consensus.cc:348] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.001417 1818 raft_consensus.cc:374] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:16.001508 1818 raft_consensus.cc:729] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed430d674974765b2c4ddf84cb77a5c, State: Initialized, Role: FOLLOWER
I20260812 16:38:16.001747 1818 consensus_queue.cc:260] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.002022 1819 tablet_bootstrap.cc:492] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: No bootstrap required, opened a new log
I20260812 16:38:16.002180 1819 ts_tablet_manager.cc:1397] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 16:38:16.002336 1818 ts_tablet_manager.cc:1428] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 16:38:16.002256 1824 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.002653 1824 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:16.002918 1819 raft_consensus.cc:348] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.003123 1819 raft_consensus.cc:374] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:16.003194 1819 raft_consensus.cc:729] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576018e1f4e84fab94f555491f3cbfe7, State: Initialized, Role: FOLLOWER
I20260812 16:38:16.003540 1819 consensus_queue.cc:260] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.004246 1819 ts_tablet_manager.cc:1428] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 16:38:16.004374 1822 tablet_bootstrap.cc:492] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: No bootstrap required, opened a new log
I20260812 16:38:16.004146 1825 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.004631 1825 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:16.004626 1822 ts_tablet_manager.cc:1397] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 16:38:16.005319 1824 raft_consensus.cc:482] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:16.005551 1824 raft_consensus.cc:504] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.006767 1822 raft_consensus.cc:348] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.007251 1822 raft_consensus.cc:374] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:16.008177 1822 raft_consensus.cc:729] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa288f8844b2d8236febe291ab53d, State: Initialized, Role: FOLLOWER
I20260812 16:38:16.008436 1822 consensus_queue.cc:260] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.008934 1826 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.009286 1826 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:16.009312 1822 ts_tablet_manager.cc:1428] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260812 16:38:16.011617 1824 leader_election.cc:288] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), e07fa288f8844b2d8236febe291ab53d (127.31.61.131:38493)
I20260812 16:38:16.019690 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.009960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:54825 (local address 127.31.61.129:45687)
0812 16:38:16.010158 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:16.010165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:16.013232 (+ 3067us) server_negotiation.cc:407] Connection header received
0812 16:38:16.013420 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:16.013425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:16.013506 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:16.013594 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:16.014372 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.015502 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.017671 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.018548 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.018721 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.018797 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:16.019210 (+ 413us) server_negotiation.cc:299] Negotiation successful
0812 16:38:16.019358 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 16:38:16.019809 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.009681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:45687 (local address 127.31.61.130:54825)
0812 16:38:16.013126 (+ 3445us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:16.013143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:16.013279 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:16.013770 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:16.013776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:16.013802 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:16.014200 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.014212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.015651 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.015657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.017495 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.017511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.018812 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.018820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.019014 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.019059 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 16:38:16.019173 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 16:38:16.019318 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3310,"mutex_wait_us":29,"spinlock_wait_cycles":21120,"thread_start_us":3200,"threads_started":1}
I20260812 16:38:16.019939 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.012662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:46559 (local address 127.31.61.131:38493)
0812 16:38:16.012865 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:16.012871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:16.013946 (+ 1075us) server_negotiation.cc:407] Connection header received
0812 16:38:16.014147 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:16.014152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:16.014231 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:16.014336 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:16.014902 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.015851 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.017739 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.018677 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.018843 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.018926 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:16.019491 (+ 565us) server_negotiation.cc:299] Negotiation successful
0812 16:38:16.019677 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 16:38:16.020576 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.012129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:38493 (local address 127.31.61.130:46559)
0812 16:38:16.013827 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:16.013848 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:16.014005 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:16.014346 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:16.014350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:16.014377 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:16.014736 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.014748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.015994 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.016000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.017567 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.017582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.019109 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.019116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.019291 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.019323 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:38:16.020111 (+ 788us) client_negotiation.cc:240] Negotiation successful
0812 16:38:16.020229 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"mutex_wait_us":1,"spinlock_wait_cycles":1408,"thread_start_us":1370,"threads_started":1}
I20260812 16:38:16.020654 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2fd4fa5bf0433ba87aad214511950c" candidate_uuid: "8ed430d674974765b2c4ddf84cb77a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7" is_pre_election: true
I20260812 16:38:16.020946 1465 raft_consensus.cc:2454] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed430d674974765b2c4ddf84cb77a5c in term 0.
I20260812 16:38:16.021411 1537 leader_election.cc:302] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576018e1f4e84fab94f555491f3cbfe7, 8ed430d674974765b2c4ddf84cb77a5c; no voters:
I20260812 16:38:16.021535 1749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2fd4fa5bf0433ba87aad214511950c" candidate_uuid: "8ed430d674974765b2c4ddf84cb77a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa288f8844b2d8236febe291ab53d" is_pre_election: true
I20260812 16:38:16.021798 1824 raft_consensus.cc:2781] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:16.021922 1824 raft_consensus.cc:482] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:16.022002 1824 raft_consensus.cc:3037] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:16.021950 1749 raft_consensus.cc:2454] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed430d674974765b2c4ddf84cb77a5c in term 0.
I20260812 16:38:16.024933 1824 raft_consensus.cc:504] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.025738 1824 leader_election.cc:288] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [CANDIDATE]: Term 1 election: Requested vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), e07fa288f8844b2d8236febe291ab53d (127.31.61.131:38493)
I20260812 16:38:16.026364 1749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2fd4fa5bf0433ba87aad214511950c" candidate_uuid: "8ed430d674974765b2c4ddf84cb77a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa288f8844b2d8236febe291ab53d"
I20260812 16:38:16.026491 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2fd4fa5bf0433ba87aad214511950c" candidate_uuid: "8ed430d674974765b2c4ddf84cb77a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7"
I20260812 16:38:16.026605 1749 raft_consensus.cc:3037] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:16.026748 1465 raft_consensus.cc:3037] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:16.030092 1465 raft_consensus.cc:2454] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed430d674974765b2c4ddf84cb77a5c in term 1.
I20260812 16:38:16.030125 1749 raft_consensus.cc:2454] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed430d674974765b2c4ddf84cb77a5c in term 1.
I20260812 16:38:16.030879 1538 leader_election.cc:302] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed430d674974765b2c4ddf84cb77a5c, e07fa288f8844b2d8236febe291ab53d; no voters:
I20260812 16:38:16.032274 1824 raft_consensus.cc:2781] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:16.032960 1824 raft_consensus.cc:686] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [term 1 LEADER]: Becoming Leader. State: Replica: 8ed430d674974765b2c4ddf84cb77a5c, State: Running, Role: LEADER
I20260812 16:38:16.033674 1824 consensus_queue.cc:237] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.037349 1842 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: TxnStatusTablet state changed. Reason: New leader 8ed430d674974765b2c4ddf84cb77a5c. Latest consensus state: current_term: 1 leader_uuid: "8ed430d674974765b2c4ddf84cb77a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.037988 1842 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:38:16.038825 1844 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 16:38:16.039076 1844 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 16:38:16.038944 1305 catalog_manager.cc:5495] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130). New cstate: current_term: 1 leader_uuid: "8ed430d674974765b2c4ddf84cb77a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } health_report { overall_health: UNKNOWN } } }
W20260812 16:38:16.059697 1517 tablet.cc:2369] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:38:16.065357 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.058505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:43197 (local address 127.0.0.1:47832)
0812 16:38:16.058745 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:16.058765 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:16.058890 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:16.059458 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:16.059463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:16.059475 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:38:16.059712 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.059724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.061213 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.061218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.062334 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.062348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.063456 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.063462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.063722 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.063791 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:38:16.064869 (+ 1078us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:38:16.064892 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 16:38:16.064987 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 16:38:16.065116 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 16:38:16.065563 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.058520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.31.61.130:43197)
0812 16:38:16.058716 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:16.058723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:16.058833 (+ 110us) server_negotiation.cc:407] Connection header received
0812 16:38:16.059099 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:16.059105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:16.059218 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:16.059413 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:16.059878 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.061044 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.062471 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.063226 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.063365 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.064002 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:16.064738 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:16.064885 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:16.065027 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 16:38:16.065131 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:38:16.074930 1749 raft_consensus.cc:1261] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Refusing update from remote peer 8ed430d674974765b2c4ddf84cb77a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:16.076472 1465 raft_consensus.cc:1261] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Refusing update from remote peer 8ed430d674974765b2c4ddf84cb77a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:16.076390 1842 consensus_queue.cc:1035] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:16.077198 1842 consensus_queue.cc:1035] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:16.084286 1826 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: TxnStatusTablet state changed. Reason: New leader 8ed430d674974765b2c4ddf84cb77a5c. Latest consensus state: current_term: 1 leader_uuid: "8ed430d674974765b2c4ddf84cb77a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.084640 1826 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:16.086781 1847 mvcc.cc:205] Tried to move back new op lower bound from 7317719843110883328 to 7317719842958438400. Current Snapshot: MvccSnapshot[applied={T|T < 7317719843110883328}]
I20260812 16:38:16.087968 1826 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ed430d674974765b2c4ddf84cb77a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.088235 1826 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P e07fa288f8844b2d8236febe291ab53d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:16.092247 1824 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ed430d674974765b2c4ddf84cb77a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.092567 1824 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:38:16.097879 1825 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: TxnStatusTablet state changed. Reason: New leader 8ed430d674974765b2c4ddf84cb77a5c. Latest consensus state: current_term: 1 leader_uuid: "8ed430d674974765b2c4ddf84cb77a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.098263 1825 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:16.100129 1842 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ed430d674974765b2c4ddf84cb77a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.100404 1842 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P 8ed430d674974765b2c4ddf84cb77a5c: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:38:16.105862 1825 tablet_replica.cc:439] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ed430d674974765b2c4ddf84cb77a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } } }
I20260812 16:38:16.106153 1825 tablet_replica.cc:442] T 7e2fd4fa5bf0433ba87aad214511950c P 576018e1f4e84fab94f555491f3cbfe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:38:16.110220 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.101428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52542 (local address 127.31.61.131:38493)
0812 16:38:16.101860 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:16.101866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:16.102016 (+ 150us) server_negotiation.cc:407] Connection header received
0812 16:38:16.102254 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:16.102259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:16.102363 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:16.102512 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:16.103831 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.104958 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.107316 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.108119 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.108269 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.108849 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:16.109557 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:16.109718 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:16.109835 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 16:38:16.109927 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260812 16:38:16.132548 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.123924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:51811 (local address 127.31.61.130:43197)
0812 16:38:16.124203 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:16.124209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:16.124433 (+ 224us) server_negotiation.cc:407] Connection header received
0812 16:38:16.124652 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:16.124657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:16.124745 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:16.124853 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:16.125286 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.126710 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.129214 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.129879 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.130002 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.131057 (+ 1055us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:16.131187 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:16.131279 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:16.132057 (+ 778us) server_negotiation.cc:299] Negotiation successful
0812 16:38:16.132245 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:38:16.132997 1767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.123935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:43197 (local address 127.31.61.131:51811)
0812 16:38:16.124302 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:16.124327 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:16.124506 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:16.124898 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:16.124904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:16.124919 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:38:16.125138 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.125151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.126906 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.126913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.128992 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.129013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.130225 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.130236 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.130448 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.130629 (+ 181us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:38:16.131387 (+ 758us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:38:16.131406 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 16:38:16.131979 (+ 573us) client_negotiation.cc:240] Negotiation successful
0812 16:38:16.132287 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260812 16:38:16.145915 1465 raft_consensus.cc:1261] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Refusing update from remote peer e07fa288f8844b2d8236febe291ab53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:16.146307 1607 raft_consensus.cc:1261] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Refusing update from remote peer e07fa288f8844b2d8236febe291ab53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:16.146627 1828 consensus_queue.cc:1035] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:16.147256 1826 consensus_queue.cc:1035] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:16.154028 1848 mvcc.cc:205] Tried to move back new op lower bound from 7317719843406569472 to 7317719842384310272. Current Snapshot: MvccSnapshot[applied={T|T < 7317719843406569472}]
I20260812 16:38:16.169965 1849 mvcc.cc:205] Tried to move back new op lower bound from 7317719843406569472 to 7317719842384310272. Current Snapshot: MvccSnapshot[applied={T|T < 7317719843406569472}]
I20260812 16:38:16.195252 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.186999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:55579 (local address 127.31.61.131:38493)
0812 16:38:16.187703 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:16.187709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:16.188778 (+ 1069us) server_negotiation.cc:407] Connection header received
0812 16:38:16.189043 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:16.189047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:16.189116 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:16.189188 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:16.189818 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.190880 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.193055 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.193578 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.193714 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.194499 (+ 785us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:16.194621 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:16.194710 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:16.194855 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 16:38:16.194965 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490}
I20260812 16:38:16.195497 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:16.187178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:38493 (local address 127.31.61.130:55579)
0812 16:38:16.188682 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:16.188701 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:16.188896 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:16.189228 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:16.189232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:16.189244 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:38:16.189618 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.189633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.191033 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.191037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.192882 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:16.192902 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:16.193796 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:16.193803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:16.194004 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:16.194124 (+ 120us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:38:16.194725 (+ 601us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:38:16.194739 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 16:38:16.194824 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 16:38:16.194991 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"thread_start_us":681,"threads_started":1}
I20260812 16:38:16.241725 1304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:16.242059 1304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:16.246781 1304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e: Sending DeleteTablet for 3 replicas of tablet 585e8694e5d541d2ab3f32272b919fa0
I20260812 16:38:16.248736 1445 tablet_service.cc:1514] Processing DeleteTablet for tablet 585e8694e5d541d2ab3f32272b919fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:16 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:35978
I20260812 16:38:16.248857 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 585e8694e5d541d2ab3f32272b919fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:16 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:47830
I20260812 16:38:16.249593 1729 tablet_service.cc:1514] Processing DeleteTablet for tablet 585e8694e5d541d2ab3f32272b919fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:16 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:52540
I20260812 16:38:16.265146 1867 tablet_replica.cc:330] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: stopping tablet replica
I20260812 16:38:16.265733 1867 raft_consensus.cc:2229] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:16.266165 1867 raft_consensus.cc:2258] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:16.269109 1867 ts_tablet_manager.cc:1905] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:16.268836 1304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:16.271847 1304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:16.278241 1867 ts_tablet_manager.cc:1918] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:38:16.278458 1867 log.cc:1196] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/585e8694e5d541d2ab3f32272b919fa0
I20260812 16:38:16.279040 1867 ts_tablet_manager.cc:1939] T 585e8694e5d541d2ab3f32272b919fa0 P 8ed430d674974765b2c4ddf84cb77a5c: Deleting consensus metadata
I20260812 16:38:16.280586 1868 tablet_replica.cc:330] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: stopping tablet replica
I20260812 16:38:16.282464 1293 catalog_manager.cc:4841] TS 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197): tablet 585e8694e5d541d2ab3f32272b919fa0 (table test-table [id=0a87310b8b0a462e953ee6aae1dc0b98]) successfully deleted
I20260812 16:38:16.285804 1586 tablet_service.cc:1467] Processing CreateTablet for tablet cbb374869c45454db7e10ca67dc7ebfa (DEFAULT_TABLE table=test-table [id=12bb03399d624a9eb31a0e48ca2d1bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:16.285245 1444 tablet_service.cc:1467] Processing CreateTablet for tablet cbb374869c45454db7e10ca67dc7ebfa (DEFAULT_TABLE table=test-table [id=12bb03399d624a9eb31a0e48ca2d1bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:16.286389 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbb374869c45454db7e10ca67dc7ebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:16.286708 1444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbb374869c45454db7e10ca67dc7ebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:16.294909 1818 tablet_bootstrap.cc:492] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: Bootstrap starting.
I20260812 16:38:16.295693 1819 tablet_bootstrap.cc:492] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: Bootstrap starting.
I20260812 16:38:16.296702 1818 tablet_bootstrap.cc:654] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:16.297318 1819 tablet_bootstrap.cc:654] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:16.299386 1818 tablet_bootstrap.cc:492] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: No bootstrap required, opened a new log
I20260812 16:38:16.299580 1818 ts_tablet_manager.cc:1397] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 16:38:16.300526 1818 raft_consensus.cc:348] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.300812 1818 raft_consensus.cc:374] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:16.300891 1818 raft_consensus.cc:729] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed430d674974765b2c4ddf84cb77a5c, State: Initialized, Role: FOLLOWER
I20260812 16:38:16.301266 1818 consensus_queue.cc:260] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.302922 1729 tablet_service.cc:1467] Processing CreateTablet for tablet cbb374869c45454db7e10ca67dc7ebfa (DEFAULT_TABLE table=test-table [id=12bb03399d624a9eb31a0e48ca2d1bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:16.303458 1729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbb374869c45454db7e10ca67dc7ebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:16.304713 1818 ts_tablet_manager.cc:1428] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260812 16:38:16.308687 1868 raft_consensus.cc:2229] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:16.309455 1868 raft_consensus.cc:2258] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:16.313746 1868 ts_tablet_manager.cc:1905] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:16.317283 1822 tablet_bootstrap.cc:492] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: Bootstrap starting.
I20260812 16:38:16.317327 1866 tablet_replica.cc:330] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: stopping tablet replica
I20260812 16:38:16.317796 1866 raft_consensus.cc:2229] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:16.318209 1866 raft_consensus.cc:2258] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:16.319104 1822 tablet_bootstrap.cc:654] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:16.320550 1866 ts_tablet_manager.cc:1905] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:16.322485 1819 tablet_bootstrap.cc:492] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: No bootstrap required, opened a new log
I20260812 16:38:16.322851 1819 ts_tablet_manager.cc:1397] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.004s
I20260812 16:38:16.323920 1819 raft_consensus.cc:348] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.324229 1819 raft_consensus.cc:374] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:16.324407 1819 raft_consensus.cc:729] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576018e1f4e84fab94f555491f3cbfe7, State: Initialized, Role: FOLLOWER
I20260812 16:38:16.324833 1819 consensus_queue.cc:260] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.330013 1868 ts_tablet_manager.cc:1918] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:38:16.330238 1868 log.cc:1196] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/585e8694e5d541d2ab3f32272b919fa0
I20260812 16:38:16.331856 1868 ts_tablet_manager.cc:1939] T 585e8694e5d541d2ab3f32272b919fa0 P e07fa288f8844b2d8236febe291ab53d: Deleting consensus metadata
I20260812 16:38:16.333153 1819 ts_tablet_manager.cc:1428] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260812 16:38:16.334151 1822 tablet_bootstrap.cc:492] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: No bootstrap required, opened a new log
I20260812 16:38:16.334118 1292 catalog_manager.cc:4841] TS e07fa288f8844b2d8236febe291ab53d (127.31.61.131:38493): tablet 585e8694e5d541d2ab3f32272b919fa0 (table test-table [id=0a87310b8b0a462e953ee6aae1dc0b98]) successfully deleted
I20260812 16:38:16.334332 1822 ts_tablet_manager.cc:1397] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260812 16:38:16.335317 1822 raft_consensus.cc:348] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.335635 1822 raft_consensus.cc:374] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:16.335741 1822 raft_consensus.cc:729] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa288f8844b2d8236febe291ab53d, State: Initialized, Role: FOLLOWER
I20260812 16:38:16.336570 1822 consensus_queue.cc:260] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.337776 1822 ts_tablet_manager.cc:1428] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 16:38:16.338611 1866 ts_tablet_manager.cc:1918] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:38:16.338801 1866 log.cc:1196] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/585e8694e5d541d2ab3f32272b919fa0
I20260812 16:38:16.339323 1866 ts_tablet_manager.cc:1939] T 585e8694e5d541d2ab3f32272b919fa0 P 576018e1f4e84fab94f555491f3cbfe7: Deleting consensus metadata
I20260812 16:38:16.341216 1291 catalog_manager.cc:4841] TS 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687): tablet 585e8694e5d541d2ab3f32272b919fa0 (table test-table [id=0a87310b8b0a462e953ee6aae1dc0b98]) successfully deleted
I20260812 16:38:16.455691 1826 raft_consensus.cc:482] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:16.455936 1826 raft_consensus.cc:504] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.456760 1826 leader_election.cc:288] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197)
I20260812 16:38:16.457376 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbb374869c45454db7e10ca67dc7ebfa" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7" is_pre_election: true
I20260812 16:38:16.457639 1465 raft_consensus.cc:2454] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 0.
I20260812 16:38:16.457595 1606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbb374869c45454db7e10ca67dc7ebfa" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed430d674974765b2c4ddf84cb77a5c" is_pre_election: true
I20260812 16:38:16.457906 1606 raft_consensus.cc:2454] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 0.
I20260812 16:38:16.458155 1679 leader_election.cc:302] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576018e1f4e84fab94f555491f3cbfe7, e07fa288f8844b2d8236febe291ab53d; no voters:
I20260812 16:38:16.458665 1826 raft_consensus.cc:2781] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:16.458806 1826 raft_consensus.cc:482] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:16.458899 1826 raft_consensus.cc:3037] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:16.462491 1826 raft_consensus.cc:504] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.463929 1826 leader_election.cc:288] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 election: Requested vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197)
I20260812 16:38:16.464504 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbb374869c45454db7e10ca67dc7ebfa" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7"
I20260812 16:38:16.464754 1465 raft_consensus.cc:3037] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:16.465121 1606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbb374869c45454db7e10ca67dc7ebfa" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed430d674974765b2c4ddf84cb77a5c"
I20260812 16:38:16.465829 1606 raft_consensus.cc:3037] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:16.467873 1465 raft_consensus.cc:2454] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 1.
I20260812 16:38:16.468463 1679 leader_election.cc:302] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576018e1f4e84fab94f555491f3cbfe7, e07fa288f8844b2d8236febe291ab53d; no voters:
I20260812 16:38:16.468966 1826 raft_consensus.cc:2781] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:16.469276 1826 raft_consensus.cc:686] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 1 LEADER]: Becoming Leader. State: Replica: e07fa288f8844b2d8236febe291ab53d, State: Running, Role: LEADER
I20260812 16:38:16.469645 1826 consensus_queue.cc:237] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:16.472819 1606 raft_consensus.cc:2454] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 1.
I20260812 16:38:16.476589 1304 catalog_manager.cc:5495] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d reported cstate change: term changed from 0 to 1, leader changed from <none> to e07fa288f8844b2d8236febe291ab53d (127.31.61.131). New cstate: current_term: 1 leader_uuid: "e07fa288f8844b2d8236febe291ab53d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } health_report { overall_health: HEALTHY } } }
I20260812 16:38:16.544812 1465 raft_consensus.cc:1261] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Refusing update from remote peer e07fa288f8844b2d8236febe291ab53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:16.545027 1606 raft_consensus.cc:1261] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Refusing update from remote peer e07fa288f8844b2d8236febe291ab53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:16.545472 1861 consensus_queue.cc:1035] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:16.545794 1828 consensus_queue.cc:1035] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:38:16.591293 1848 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 16:38:18.198808 1304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:18.199069 1304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:18.203454 1304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e: Sending DeleteTablet for 3 replicas of tablet cbb374869c45454db7e10ca67dc7ebfa
I20260812 16:38:18.204385 1445 tablet_service.cc:1514] Processing DeleteTablet for tablet cbb374869c45454db7e10ca67dc7ebfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:18 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:35978
I20260812 16:38:18.204429 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet cbb374869c45454db7e10ca67dc7ebfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:18 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:47830
I20260812 16:38:18.204499 1729 tablet_service.cc:1514] Processing DeleteTablet for tablet cbb374869c45454db7e10ca67dc7ebfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:18 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:52540
I20260812 16:38:18.216794 1902 tablet_replica.cc:330] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: stopping tablet replica
I20260812 16:38:18.217020 1901 tablet_replica.cc:330] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: stopping tablet replica
I20260812 16:38:18.217052 1902 raft_consensus.cc:2229] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:18.217280 1901 raft_consensus.cc:2229] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:18.217517 1901 raft_consensus.cc:2258] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:18.218473 1900 tablet_replica.cc:330] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: stopping tablet replica
I20260812 16:38:18.218528 1902 raft_consensus.cc:2258] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:18.218675 1901 ts_tablet_manager.cc:1905] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:18.218726 1900 raft_consensus.cc:2229] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:18.219004 1900 raft_consensus.cc:2258] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:18.220021 1900 ts_tablet_manager.cc:1905] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:18.220104 1902 ts_tablet_manager.cc:1905] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:18.225817 1901 ts_tablet_manager.cc:1918] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:38:18.226115 1901 log.cc:1196] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/cbb374869c45454db7e10ca67dc7ebfa
I20260812 16:38:18.226570 1902 ts_tablet_manager.cc:1918] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:38:18.226825 1901 ts_tablet_manager.cc:1939] T cbb374869c45454db7e10ca67dc7ebfa P 8ed430d674974765b2c4ddf84cb77a5c: Deleting consensus metadata
I20260812 16:38:18.226835 1902 log.cc:1196] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/cbb374869c45454db7e10ca67dc7ebfa
I20260812 16:38:18.227480 1902 ts_tablet_manager.cc:1939] T cbb374869c45454db7e10ca67dc7ebfa P e07fa288f8844b2d8236febe291ab53d: Deleting consensus metadata
I20260812 16:38:18.227645 1900 ts_tablet_manager.cc:1918] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:38:18.227864 1900 log.cc:1196] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/cbb374869c45454db7e10ca67dc7ebfa
I20260812 16:38:18.228374 1293 catalog_manager.cc:4841] TS 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197): tablet cbb374869c45454db7e10ca67dc7ebfa (table test-table [id=12bb03399d624a9eb31a0e48ca2d1bfe]) successfully deleted
I20260812 16:38:18.228654 1900 ts_tablet_manager.cc:1939] T cbb374869c45454db7e10ca67dc7ebfa P 576018e1f4e84fab94f555491f3cbfe7: Deleting consensus metadata
I20260812 16:38:18.228875 1292 catalog_manager.cc:4841] TS e07fa288f8844b2d8236febe291ab53d (127.31.61.131:38493): tablet cbb374869c45454db7e10ca67dc7ebfa (table test-table [id=12bb03399d624a9eb31a0e48ca2d1bfe]) successfully deleted
I20260812 16:38:18.230219 1291 catalog_manager.cc:4841] TS 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687): tablet cbb374869c45454db7e10ca67dc7ebfa (table test-table [id=12bb03399d624a9eb31a0e48ca2d1bfe]) successfully deleted
Aug 12 16:38:18 dist-test-slave-f7th krb5kdc[1260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:38:18 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:18.272763 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:18.253740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.31.61.190:45007)
0812 16:38:18.254298 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:18.254304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:18.254324 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:38:18.254379 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:18.254383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:18.254444 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:18.254539 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:18.256031 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:18.256791 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:18.258049 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:18.258500 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:18.258620 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:18.264342 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:18.264348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:18.264352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:18.264401 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:18.268941 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:18.270073 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:18.270078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:18.270080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:18.270228 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:18.270957 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:18.270962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:18.270964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:18.271408 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:18.271853 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:18.272180 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 16:38:18.272389 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":253,"threads_started":1}
I20260812 16:38:18.276925 1304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:18.277578 1304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:18.284680 1445 tablet_service.cc:1467] Processing CreateTablet for tablet 07af5998e22f4ec590924632a3ec8850 (DEFAULT_TABLE table=test-table [id=eac5f5ab8bef48f6895ec5a0e8bd23bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:18.284894 1586 tablet_service.cc:1467] Processing CreateTablet for tablet 07af5998e22f4ec590924632a3ec8850 (DEFAULT_TABLE table=test-table [id=eac5f5ab8bef48f6895ec5a0e8bd23bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:18.285012 1729 tablet_service.cc:1467] Processing CreateTablet for tablet 07af5998e22f4ec590924632a3ec8850 (DEFAULT_TABLE table=test-table [id=eac5f5ab8bef48f6895ec5a0e8bd23bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:18.285115 1445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07af5998e22f4ec590924632a3ec8850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:18.285323 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07af5998e22f4ec590924632a3ec8850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:18.285392 1729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07af5998e22f4ec590924632a3ec8850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:18.290843 1911 tablet_bootstrap.cc:492] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: Bootstrap starting.
I20260812 16:38:18.292265 1912 tablet_bootstrap.cc:492] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: Bootstrap starting.
I20260812 16:38:18.292680 1911 tablet_bootstrap.cc:654] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:18.294193 1912 tablet_bootstrap.cc:654] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:18.296857 1911 tablet_bootstrap.cc:492] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: No bootstrap required, opened a new log
I20260812 16:38:18.297129 1911 ts_tablet_manager.cc:1397] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260812 16:38:18.297416 1912 tablet_bootstrap.cc:492] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: No bootstrap required, opened a new log
I20260812 16:38:18.297676 1912 ts_tablet_manager.cc:1397] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260812 16:38:18.298255 1911 raft_consensus.cc:348] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.298552 1911 raft_consensus.cc:374] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:18.298403 1912 raft_consensus.cc:348] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.298663 1911 raft_consensus.cc:729] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576018e1f4e84fab94f555491f3cbfe7, State: Initialized, Role: FOLLOWER
I20260812 16:38:18.298753 1912 raft_consensus.cc:374] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:18.298933 1912 raft_consensus.cc:729] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa288f8844b2d8236febe291ab53d, State: Initialized, Role: FOLLOWER
I20260812 16:38:18.299067 1911 consensus_queue.cc:260] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.299211 1912 consensus_queue.cc:260] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.300005 1912 ts_tablet_manager.cc:1428] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 16:38:18.300139 1913 tablet_bootstrap.cc:492] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: Bootstrap starting.
I20260812 16:38:18.301355 1911 ts_tablet_manager.cc:1428] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260812 16:38:18.302342 1913 tablet_bootstrap.cc:654] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:18.305372 1913 tablet_bootstrap.cc:492] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: No bootstrap required, opened a new log
I20260812 16:38:18.305521 1913 ts_tablet_manager.cc:1397] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 16:38:18.306144 1913 raft_consensus.cc:348] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.306382 1913 raft_consensus.cc:374] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:18.306479 1913 raft_consensus.cc:729] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed430d674974765b2c4ddf84cb77a5c, State: Initialized, Role: FOLLOWER
I20260812 16:38:18.306713 1913 consensus_queue.cc:260] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.307415 1913 ts_tablet_manager.cc:1428] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 16:38:18.352834 1877 raft_consensus.cc:482] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:18.353039 1877 raft_consensus.cc:504] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.353735 1877 leader_election.cc:288] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197)
I20260812 16:38:18.354456 1607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07af5998e22f4ec590924632a3ec8850" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed430d674974765b2c4ddf84cb77a5c" is_pre_election: true
I20260812 16:38:18.354697 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07af5998e22f4ec590924632a3ec8850" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7" is_pre_election: true
I20260812 16:38:18.354846 1607 raft_consensus.cc:2454] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 0.
I20260812 16:38:18.355027 1465 raft_consensus.cc:2454] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 0.
I20260812 16:38:18.355538 1681 leader_election.cc:302] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed430d674974765b2c4ddf84cb77a5c, e07fa288f8844b2d8236febe291ab53d; no voters:
I20260812 16:38:18.355983 1877 raft_consensus.cc:2781] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:18.356113 1877 raft_consensus.cc:482] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:18.356222 1877 raft_consensus.cc:3037] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:18.358263 1877 raft_consensus.cc:504] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.358875 1877 leader_election.cc:288] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 election: Requested vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197)
I20260812 16:38:18.359408 1607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07af5998e22f4ec590924632a3ec8850" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed430d674974765b2c4ddf84cb77a5c"
I20260812 16:38:18.359416 1465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07af5998e22f4ec590924632a3ec8850" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7"
I20260812 16:38:18.359661 1465 raft_consensus.cc:3037] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:18.359660 1607 raft_consensus.cc:3037] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:18.362788 1607 raft_consensus.cc:2454] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 1.
I20260812 16:38:18.362788 1465 raft_consensus.cc:2454] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 1.
I20260812 16:38:18.363440 1679 leader_election.cc:302] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576018e1f4e84fab94f555491f3cbfe7, e07fa288f8844b2d8236febe291ab53d; no voters:
I20260812 16:38:18.363842 1877 raft_consensus.cc:2781] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:18.364063 1877 raft_consensus.cc:686] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 1 LEADER]: Becoming Leader. State: Replica: e07fa288f8844b2d8236febe291ab53d, State: Running, Role: LEADER
I20260812 16:38:18.364359 1877 consensus_queue.cc:237] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.367244 1304 catalog_manager.cc:5495] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d reported cstate change: term changed from 0 to 1, leader changed from <none> to e07fa288f8844b2d8236febe291ab53d (127.31.61.131). New cstate: current_term: 1 leader_uuid: "e07fa288f8844b2d8236febe291ab53d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } health_report { overall_health: HEALTHY } } }
I20260812 16:38:18.394207 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:18.388460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.31.61.131:38493)
0812 16:38:18.389040 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:18.389044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:18.389064 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:38:18.389120 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:18.389123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:18.389183 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:18.389269 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:18.389554 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:18.390488 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:18.391856 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:18.392519 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:18.392715 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:18.393414 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:18.393542 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:18.393618 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:18.393830 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 16:38:18.393900 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":333,"threads_started":1}
I20260812 16:38:18.404163 1465 raft_consensus.cc:1261] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Refusing update from remote peer e07fa288f8844b2d8236febe291ab53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:18.404265 1607 raft_consensus.cc:1261] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Refusing update from remote peer e07fa288f8844b2d8236febe291ab53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:18.404806 1877 consensus_queue.cc:1035] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:18.405076 1890 consensus_queue.cc:1035] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:18.448045 1304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:18.448295 1304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:18.451747 1304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e: Sending DeleteTablet for 3 replicas of tablet 07af5998e22f4ec590924632a3ec8850
I20260812 16:38:18.452586 1445 tablet_service.cc:1514] Processing DeleteTablet for tablet 07af5998e22f4ec590924632a3ec8850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:18 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:35978
I20260812 16:38:18.452661 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 07af5998e22f4ec590924632a3ec8850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:18 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:47830
I20260812 16:38:18.452798 1729 tablet_service.cc:1514] Processing DeleteTablet for tablet 07af5998e22f4ec590924632a3ec8850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:18 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:52540
I20260812 16:38:18.452925 1901 tablet_replica.cc:330] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: stopping tablet replica
I20260812 16:38:18.453115 1901 raft_consensus.cc:2229] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:18.453145 1900 tablet_replica.cc:330] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: stopping tablet replica
I20260812 16:38:18.453298 1901 raft_consensus.cc:2258] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:18.453408 1900 raft_consensus.cc:2229] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:18.453593 1902 tablet_replica.cc:330] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: stopping tablet replica
I20260812 16:38:18.453635 1900 raft_consensus.cc:2258] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:18.453787 1902 raft_consensus.cc:2229] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:18.453903 1304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 16:38:18.454155 1902 raft_consensus.cc:2258] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 16:38:18.454524 1304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:18.454768 1901 ts_tablet_manager.cc:1905] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:18.454768 1900 ts_tablet_manager.cc:1905] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:18.460772 1729 tablet_service.cc:1467] Processing CreateTablet for tablet f57f0b6ce4c84c678b77df9df46270ed (DEFAULT_TABLE table=test-table [id=f136ef7ca6e04faeb0ce01e6ab61fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:18.460781 1586 tablet_service.cc:1467] Processing CreateTablet for tablet f57f0b6ce4c84c678b77df9df46270ed (DEFAULT_TABLE table=test-table [id=f136ef7ca6e04faeb0ce01e6ab61fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:18.461231 1729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57f0b6ce4c84c678b77df9df46270ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:18.461236 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57f0b6ce4c84c678b77df9df46270ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:18.462173 1900 ts_tablet_manager.cc:1918] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:38:18.462366 1900 log.cc:1196] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/07af5998e22f4ec590924632a3ec8850
I20260812 16:38:18.462834 1901 ts_tablet_manager.cc:1918] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:38:18.462989 1900 ts_tablet_manager.cc:1939] T 07af5998e22f4ec590924632a3ec8850 P 576018e1f4e84fab94f555491f3cbfe7: Deleting consensus metadata
I20260812 16:38:18.463131 1901 log.cc:1196] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/07af5998e22f4ec590924632a3ec8850
I20260812 16:38:18.463696 1901 ts_tablet_manager.cc:1939] T 07af5998e22f4ec590924632a3ec8850 P 8ed430d674974765b2c4ddf84cb77a5c: Deleting consensus metadata
I20260812 16:38:18.464270 1291 catalog_manager.cc:4841] TS 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687): tablet 07af5998e22f4ec590924632a3ec8850 (table test-table [id=eac5f5ab8bef48f6895ec5a0e8bd23bd]) successfully deleted
I20260812 16:38:18.464778 1445 tablet_service.cc:1467] Processing CreateTablet for tablet f57f0b6ce4c84c678b77df9df46270ed (DEFAULT_TABLE table=test-table [id=f136ef7ca6e04faeb0ce01e6ab61fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:18.465191 1445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57f0b6ce4c84c678b77df9df46270ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:18.465242 1902 ts_tablet_manager.cc:1905] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:18.469149 1912 tablet_bootstrap.cc:492] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: Bootstrap starting.
I20260812 16:38:18.470997 1912 tablet_bootstrap.cc:654] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:18.472812 1902 ts_tablet_manager.cc:1918] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:38:18.473026 1902 log.cc:1196] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/07af5998e22f4ec590924632a3ec8850
I20260812 16:38:18.473003 1293 catalog_manager.cc:4841] TS 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197): tablet 07af5998e22f4ec590924632a3ec8850 (table test-table [id=eac5f5ab8bef48f6895ec5a0e8bd23bd]) successfully deleted
I20260812 16:38:18.473490 1902 ts_tablet_manager.cc:1939] T 07af5998e22f4ec590924632a3ec8850 P e07fa288f8844b2d8236febe291ab53d: Deleting consensus metadata
I20260812 16:38:18.474885 1292 catalog_manager.cc:4841] TS e07fa288f8844b2d8236febe291ab53d (127.31.61.131:38493): tablet 07af5998e22f4ec590924632a3ec8850 (table test-table [id=eac5f5ab8bef48f6895ec5a0e8bd23bd]) successfully deleted
I20260812 16:38:18.477041 1911 tablet_bootstrap.cc:492] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: Bootstrap starting.
I20260812 16:38:18.478889 1913 tablet_bootstrap.cc:492] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: Bootstrap starting.
I20260812 16:38:18.479117 1911 tablet_bootstrap.cc:654] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:18.480755 1912 tablet_bootstrap.cc:492] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: No bootstrap required, opened a new log
I20260812 16:38:18.480829 1913 tablet_bootstrap.cc:654] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:18.480960 1912 ts_tablet_manager.cc:1397] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260812 16:38:18.481822 1912 raft_consensus.cc:348] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.482082 1912 raft_consensus.cc:374] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:18.482167 1912 raft_consensus.cc:729] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa288f8844b2d8236febe291ab53d, State: Initialized, Role: FOLLOWER
I20260812 16:38:18.482427 1912 consensus_queue.cc:260] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.483129 1912 ts_tablet_manager.cc:1428] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260812 16:38:18.484393 1913 tablet_bootstrap.cc:492] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: No bootstrap required, opened a new log
I20260812 16:38:18.484571 1913 ts_tablet_manager.cc:1397] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 16:38:18.484831 1911 tablet_bootstrap.cc:492] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: No bootstrap required, opened a new log
I20260812 16:38:18.485001 1911 ts_tablet_manager.cc:1397] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260812 16:38:18.485378 1913 raft_consensus.cc:348] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.485623 1913 raft_consensus.cc:374] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:18.485715 1913 raft_consensus.cc:729] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed430d674974765b2c4ddf84cb77a5c, State: Initialized, Role: FOLLOWER
I20260812 16:38:18.485744 1911 raft_consensus.cc:348] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.485982 1911 raft_consensus.cc:374] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:18.486061 1911 raft_consensus.cc:729] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576018e1f4e84fab94f555491f3cbfe7, State: Initialized, Role: FOLLOWER
I20260812 16:38:18.485982 1913 consensus_queue.cc:260] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.486284 1911 consensus_queue.cc:260] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.486528 1913 ts_tablet_manager.cc:1428] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 16:38:18.486817 1911 ts_tablet_manager.cc:1428] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 16:38:18.553027 1925 raft_consensus.cc:482] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:18.553225 1925 raft_consensus.cc:504] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.553874 1925 leader_election.cc:288] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197)
I20260812 16:38:18.554430 1464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57f0b6ce4c84c678b77df9df46270ed" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7" is_pre_election: true
I20260812 16:38:18.554473 1607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57f0b6ce4c84c678b77df9df46270ed" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed430d674974765b2c4ddf84cb77a5c" is_pre_election: true
I20260812 16:38:18.554694 1464 raft_consensus.cc:2454] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 0.
I20260812 16:38:18.554768 1607 raft_consensus.cc:2454] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 0.
I20260812 16:38:18.555130 1679 leader_election.cc:302] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576018e1f4e84fab94f555491f3cbfe7, e07fa288f8844b2d8236febe291ab53d; no voters:
I20260812 16:38:18.555394 1925 raft_consensus.cc:2781] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:18.555511 1925 raft_consensus.cc:482] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:18.555634 1925 raft_consensus.cc:3037] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:18.557644 1925 raft_consensus.cc:504] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.558164 1925 leader_election.cc:288] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 election: Requested vote from peers 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687), 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197)
I20260812 16:38:18.558650 1464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57f0b6ce4c84c678b77df9df46270ed" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576018e1f4e84fab94f555491f3cbfe7"
I20260812 16:38:18.558640 1607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57f0b6ce4c84c678b77df9df46270ed" candidate_uuid: "e07fa288f8844b2d8236febe291ab53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed430d674974765b2c4ddf84cb77a5c"
I20260812 16:38:18.558801 1464 raft_consensus.cc:3037] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:18.558821 1607 raft_consensus.cc:3037] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:18.560969 1464 raft_consensus.cc:2454] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 1.
I20260812 16:38:18.561347 1679 leader_election.cc:302] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576018e1f4e84fab94f555491f3cbfe7, e07fa288f8844b2d8236febe291ab53d; no voters:
I20260812 16:38:18.561614 1925 raft_consensus.cc:2781] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:18.561777 1925 raft_consensus.cc:686] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 1 LEADER]: Becoming Leader. State: Replica: e07fa288f8844b2d8236febe291ab53d, State: Running, Role: LEADER
I20260812 16:38:18.561818 1607 raft_consensus.cc:2454] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa288f8844b2d8236febe291ab53d in term 1.
I20260812 16:38:18.562419 1925 consensus_queue.cc:237] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } }
I20260812 16:38:18.564862 1305 catalog_manager.cc:5495] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d reported cstate change: term changed from 0 to 1, leader changed from <none> to e07fa288f8844b2d8236febe291ab53d (127.31.61.131). New cstate: current_term: 1 leader_uuid: "e07fa288f8844b2d8236febe291ab53d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa288f8844b2d8236febe291ab53d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38493 } health_report { overall_health: HEALTHY } } }
I20260812 16:38:18.592396 1607 raft_consensus.cc:1261] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Refusing update from remote peer e07fa288f8844b2d8236febe291ab53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:18.592388 1464 raft_consensus.cc:1261] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Refusing update from remote peer e07fa288f8844b2d8236febe291ab53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:18.592993 1890 consensus_queue.cc:1035] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed430d674974765b2c4ddf84cb77a5c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:18.593302 1925 consensus_queue.cc:1035] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "576018e1f4e84fab94f555491f3cbfe7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:38:18.614215 1881 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 16:38:20.201115 1306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:20.201351 1306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:20.205579 1306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07263668d7c14bf592cfeef03f51ab4e: Sending DeleteTablet for 3 replicas of tablet f57f0b6ce4c84c678b77df9df46270ed
I20260812 16:38:20.206413 1445 tablet_service.cc:1514] Processing DeleteTablet for tablet f57f0b6ce4c84c678b77df9df46270ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:20 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:35978
I20260812 16:38:20.206413 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet f57f0b6ce4c84c678b77df9df46270ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:20 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:47830
I20260812 16:38:20.206490 1729 tablet_service.cc:1514] Processing DeleteTablet for tablet f57f0b6ce4c84c678b77df9df46270ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:20 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:52540
I20260812 16:38:20.207074 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1378
I20260812 16:38:20.220698 1953 tablet_replica.cc:330] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: stopping tablet replica
I20260812 16:38:20.221374 1952 tablet_replica.cc:330] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: stopping tablet replica
I20260812 16:38:20.221678 1952 raft_consensus.cc:2229] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:20.221807 1954 tablet_replica.cc:330] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: stopping tablet replica
I20260812 16:38:20.221802 1953 raft_consensus.cc:2229] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:20.222059 1952 raft_consensus.cc:2258] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:20.222059 1953 raft_consensus.cc:2258] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:20.222044 1954 raft_consensus.cc:2229] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:20.222802 1954 raft_consensus.cc:2258] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:20.223141 1953 ts_tablet_manager.cc:1905] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:20.223477 1952 ts_tablet_manager.cc:1905] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:20.223814 1954 ts_tablet_manager.cc:1905] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:20.230127 1953 ts_tablet_manager.cc:1918] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:38:20.230340 1953 log.cc:1196] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/f57f0b6ce4c84c678b77df9df46270ed
I20260812 16:38:20.230355 1952 ts_tablet_manager.cc:1918] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:38:20.230474 1952 log.cc:1196] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/f57f0b6ce4c84c678b77df9df46270ed
I20260812 16:38:20.230551 1954 ts_tablet_manager.cc:1918] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 16:38:20.230767 1954 log.cc:1196] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/f57f0b6ce4c84c678b77df9df46270ed
I20260812 16:38:20.230890 1952 ts_tablet_manager.cc:1939] T f57f0b6ce4c84c678b77df9df46270ed P 8ed430d674974765b2c4ddf84cb77a5c: Deleting consensus metadata
I20260812 16:38:20.231089 1953 ts_tablet_manager.cc:1939] T f57f0b6ce4c84c678b77df9df46270ed P 576018e1f4e84fab94f555491f3cbfe7: Deleting consensus metadata
I20260812 16:38:20.231323 1954 ts_tablet_manager.cc:1939] T f57f0b6ce4c84c678b77df9df46270ed P e07fa288f8844b2d8236febe291ab53d: Deleting consensus metadata
Aug 12 16:38:20 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:20.232092 1293 catalog_manager.cc:4841] TS 8ed430d674974765b2c4ddf84cb77a5c (127.31.61.130:43197): tablet f57f0b6ce4c84c678b77df9df46270ed (table test-table [id=f136ef7ca6e04faeb0ce01e6ab61fc07]) successfully deleted
I20260812 16:38:20.232344 1291 catalog_manager.cc:4841] TS 576018e1f4e84fab94f555491f3cbfe7 (127.31.61.129:45687): tablet f57f0b6ce4c84c678b77df9df46270ed (table test-table [id=f136ef7ca6e04faeb0ce01e6ab61fc07]) successfully deleted
I20260812 16:38:20.232468 1292 catalog_manager.cc:4841] TS e07fa288f8844b2d8236febe291ab53d (127.31.61.131:38493): tablet f57f0b6ce4c84c678b77df9df46270ed (table test-table [id=f136ef7ca6e04faeb0ce01e6ab61fc07]) successfully deleted
I20260812 16:38:20.239410 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:20.208792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.31.61.129:45687)
0812 16:38:20.221573 (+ 12781us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:20.221578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:20.221607 (+ 29us) server_negotiation.cc:407] Connection header received
0812 16:38:20.221673 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:20.221678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:20.221757 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:20.221886 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:20.223749 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:20.224955 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:20.226752 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:20.227383 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:20.227529 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:20.233028 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:20.233044 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:20.233062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:20.233100 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:20.236446 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:20.237198 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:20.237206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:20.237208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:20.237369 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:20.237863 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:20.237867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:20.237869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:20.238274 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:20.238523 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:20.238938 (+ 415us) server_negotiation.cc:299] Negotiation successful
0812 16:38:20.239135 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12639,"thread_start_us":12509,"threads_started":1}
W20260812 16:38:20.240103 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35986
W20260812 16:38:20.240710 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1378: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:38:20.240902 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 1378
I20260812 16:38:20.257391 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1520
Aug 12 16:38:20 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:38:20.272585 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:20.258133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.31.61.130:43197)
0812 16:38:20.258626 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:20.258631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:20.258645 (+ 14us) server_negotiation.cc:407] Connection header received
0812 16:38:20.258696 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:20.258700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:20.258756 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:20.258846 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:20.260381 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:20.261146 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:20.262421 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:20.262887 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:20.263011 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:20.267020 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:20.267034 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:20.267050 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:20.267086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:20.269909 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:20.270582 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:20.270589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:20.270592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:20.270722 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:20.271070 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:20.271074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:20.271076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:20.271423 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:20.271719 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:20.272018 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 16:38:20.272236 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
W20260812 16:38:20.273329 1653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47846
W20260812 16:38:20.273962 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:38:20.274106 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 1520
I20260812 16:38:20.294216 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1662
Aug 12 16:38:20 dist-test-slave-f7th krb5kdc[1260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:38:20.311121 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:20.295016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.31.61.131:38493)
0812 16:38:20.295541 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:20.295546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:20.295561 (+ 15us) server_negotiation.cc:407] Connection header received
0812 16:38:20.295640 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:20.295646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:20.295707 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:20.295804 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:20.297098 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:20.297860 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:20.299036 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:20.299437 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:20.299539 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:20.304225 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:20.304245 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:20.304266 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:20.304317 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:20.308174 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:20.309048 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:20.309059 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:20.309063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:20.309277 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:20.309675 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:20.309679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:20.309682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:20.310109 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:20.310378 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:20.310617 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 16:38:20.310812 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":272,"threads_started":1}
W20260812 16:38:20.311895 1794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52564
W20260812 16:38:20.312556 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1662: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:38:20.312728 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 1662
I20260812 16:38:20.332269 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1273
I20260812 16:38:20.333460 1346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:20.450623 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 1273
2026-08-12T16:38:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9444 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:20 dist-test-slave-f7th krb5kdc[1962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:20 dist-test-slave-f7th krb5kdc[1962](info): set up 2 sockets
Aug 12 16:38:20 dist-test-slave-f7th krb5kdc[1962](info): commencing operation
krb5kdc: starting...
W20260812 16:38:21.467989 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:21 dist-test-slave-f7th krb5kdc[1962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:38:21.670849 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:39315
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:39315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:21.991637 1976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:21.991935 1976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:21.992072 1976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:22.000533 1976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:22.000651 1976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:22.000707 1976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:22.000784 1976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:22.000837 1976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:22.010906 1976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:39315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:39315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:22.012342 1976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:22.013648 1976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:22.026011 1981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:22.026003 1982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:22.026190 1984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:22.026479 1976 server_base.cc:1034] running on GCE node
I20260812 16:38:22.027661 1976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:22.029217 1976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:22.030467 1976 hybrid_clock.cc:648] HybridClock initialized: now 1786552702030391 us; error 61 us; skew 500 ppm
I20260812 16:38:22.030860 1976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:22 dist-test-slave-f7th krb5kdc[1962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:22.034268 1976 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:38:22.035887 1976 webserver.cc:460] Webserver started at http://127.31.61.190:35491/ using document root <none> and password file <none>
I20260812 16:38:22.036604 1976 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:22.036729 1976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:22.037050 1976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:22.040520 1976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "712bf47112004ca4be8c838df2cde478"
format_stamp: "Formatted at 2026-08-12 16:38:22 on dist-test-slave-f7th"
server_key: "e5f467ea105ea3187ff3f44a216d66ac"
server_key_iv: "1a8abfeb701aeccb069b7cca86034ea9"
server_key_version: "encryptionkey@0"
I20260812 16:38:22.041582 1976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "712bf47112004ca4be8c838df2cde478"
format_stamp: "Formatted at 2026-08-12 16:38:22 on dist-test-slave-f7th"
server_key: "e5f467ea105ea3187ff3f44a216d66ac"
server_key_iv: "1a8abfeb701aeccb069b7cca86034ea9"
server_key_version: "encryptionkey@0"
I20260812 16:38:22.047438 1976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:38:22.051131 1991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:22.052464 1976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:22.052623 1976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "712bf47112004ca4be8c838df2cde478"
format_stamp: "Formatted at 2026-08-12 16:38:22 on dist-test-slave-f7th"
server_key: "e5f467ea105ea3187ff3f44a216d66ac"
server_key_iv: "1a8abfeb701aeccb069b7cca86034ea9"
server_key_version: "encryptionkey@0"
I20260812 16:38:22.052817 1976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:22.079416 1976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:22.083578 1976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:22.083870 1976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:22.100761 1976 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:39315
I20260812 16:38:22.100781 2042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:39315 every 8 connection(s)
I20260812 16:38:22.102294 1976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:22.106088 2043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:22.110723 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 1976
I20260812 16:38:22.110955 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:22.111743 31990 external_mini_cluster.cc:1428] Setting key cfde4dc03a74893255d9de600b474c86
I20260812 16:38:22.117893 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478: Bootstrap starting.
I20260812 16:38:22.121723 2043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:22.123031 2043 log.cc:824] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:22.126204 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478: No bootstrap required, opened a new log
Aug 12 16:38:22 dist-test-slave-f7th krb5kdc[1962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:22.136461 2043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712bf47112004ca4be8c838df2cde478" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39315 } }
I20260812 16:38:22.136943 2043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:22.137051 2043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 712bf47112004ca4be8c838df2cde478, State: Initialized, Role: FOLLOWER
I20260812 16:38:22.137293 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:22.114037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.31.61.190:39315)
0812 16:38:22.114891 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:22.114903 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:22.114967 (+ 64us) server_negotiation.cc:407] Connection header received
0812 16:38:22.116903 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:22.116928 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:22.117361 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:22.117708 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:22.119632 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:22.121078 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:22.122859 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:22.123761 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:22.123983 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:22.127826 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:22.127839 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:22.127859 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:22.127916 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:22.132155 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:22.133018 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:22.133028 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:22.133040 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:22.133293 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:22.133791 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:22.133797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:22.133800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:22.134624 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:22.134954 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:22.135395 (+ 441us) server_negotiation.cc:299] Negotiation successful
0812 16:38:22.135887 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":378,"threads_started":1}
I20260812 16:38:22.137765 2043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712bf47112004ca4be8c838df2cde478" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39315 } }
I20260812 16:38:22.138055 2043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:22.138211 2043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:22.138382 2043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:22.141018 2043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712bf47112004ca4be8c838df2cde478" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39315 } }
I20260812 16:38:22.141499 2043 leader_election.cc:302] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 712bf47112004ca4be8c838df2cde478; no voters:
I20260812 16:38:22.142019 2043 leader_election.cc:288] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:22.142163 2048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:22.142854 2048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [term 1 LEADER]: Becoming Leader. State: Replica: 712bf47112004ca4be8c838df2cde478, State: Running, Role: LEADER
I20260812 16:38:22.143369 2048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712bf47112004ca4be8c838df2cde478" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39315 } }
I20260812 16:38:22.144066 2043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:22.146291 2050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 712bf47112004ca4be8c838df2cde478. Latest consensus state: current_term: 1 leader_uuid: "712bf47112004ca4be8c838df2cde478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712bf47112004ca4be8c838df2cde478" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39315 } } }
I20260812 16:38:22.146693 2050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:22.146605 2049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "712bf47112004ca4be8c838df2cde478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "712bf47112004ca4be8c838df2cde478" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39315 } } }
I20260812 16:38:22.146900 2049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:22.147854 2054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:22.159353 2054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:22.168437 2054 catalog_manager.cc:1347] Generated new cluster ID: a7ac16846dad45fd98d82cd50f594b36
I20260812 16:38:22.168584 2054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:22.188323 2054 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:22.190171 2054 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:22.199697 2054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 712bf47112004ca4be8c838df2cde478: Generated new TSK 0
I20260812 16:38:22.200830 2054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:38:22.251734 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39315
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:22.602294 2069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:22.602630 2069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:22.602767 2069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:22.610431 2069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:22.610572 2069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:22.610742 2069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:22.624908 2069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:22.626332 2069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:22.627779 2069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:22.643625 2074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:22.643584 2077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:22.643829 2069 server_base.cc:1034] running on GCE node
W20260812 16:38:22.643652 2075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:22.645169 2069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:22.646404 2069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:22.647855 2069 hybrid_clock.cc:648] HybridClock initialized: now 1786552702647732 us; error 48 us; skew 500 ppm
I20260812 16:38:22.648381 2069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:22 dist-test-slave-f7th krb5kdc[1962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:22.651859 2069 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:38:22.653795 2069 webserver.cc:460] Webserver started at http://127.31.61.129:42729/ using document root <none> and password file <none>
I20260812 16:38:22.654522 2069 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:22.654636 2069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:22.654935 2069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:22.658275 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "30e45d8000b8406bb608abf2b51687d9"
format_stamp: "Formatted at 2026-08-12 16:38:22 on dist-test-slave-f7th"
server_key: "f8cb6300415a4b8a4f2945cff9f1ba5d"
server_key_iv: "7dba0ebb203484b0c1fa80a3054d4f47"
server_key_version: "encryptionkey@0"
I20260812 16:38:22.659098 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "30e45d8000b8406bb608abf2b51687d9"
format_stamp: "Formatted at 2026-08-12 16:38:22 on dist-test-slave-f7th"
server_key: "f8cb6300415a4b8a4f2945cff9f1ba5d"
server_key_iv: "7dba0ebb203484b0c1fa80a3054d4f47"
server_key_version: "encryptionkey@0"
I20260812 16:38:22.665081 2069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 16:38:22.669250 2084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:22.670902 2069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 16:38:22.671103 2069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "30e45d8000b8406bb608abf2b51687d9"
format_stamp: "Formatted at 2026-08-12 16:38:22 on dist-test-slave-f7th"
server_key: "f8cb6300415a4b8a4f2945cff9f1ba5d"
server_key_iv: "7dba0ebb203484b0c1fa80a3054d4f47"
server_key_version: "encryptionkey@0"
I20260812 16:38:22.671317 2069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:22.689270 2069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:22.696079 2069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:22.696390 2069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:22.697469 2069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:22.699095 2069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:22.699194 2069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:22.699316 2069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:22.699378 2069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:22.731842 2069 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:34191
I20260812 16:38:22.731873 2198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:34191 every 8 connection(s)
I20260812 16:38:22.733106 2069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:22.737627 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 2069
I20260812 16:38:22.737833 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:22.738417 31990 external_mini_cluster.cc:1428] Setting key d2e1492a6b7061a065036fe5d3db9077
Aug 12 16:38:22 dist-test-slave-f7th krb5kdc[1962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:22.752696 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:22.735558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:40339 (local address 127.31.61.190:39315)
0812 16:38:22.736004 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:22.736009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:22.736791 (+ 782us) server_negotiation.cc:407] Connection header received
0812 16:38:22.738680 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:22.738686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:22.738780 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:22.738877 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:22.740433 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:22.741167 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:22.742716 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:22.743119 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:22.743237 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:22.746818 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:22.746823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:22.746826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:22.746859 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:22.749519 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:22.750288 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:22.750292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:22.750294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:22.750433 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:22.750865 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:22.750869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:22.750871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:22.751155 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:22.751294 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:22.752088 (+ 794us) server_negotiation.cc:299] Negotiation successful
0812 16:38:22.752327 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":219,"threads_started":1}
I20260812 16:38:22.753885 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:22.735964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:39315 (local address 127.31.61.129:40339)
0812 16:38:22.736642 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:22.736679 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:22.738402 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:22.739130 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:22.739139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:22.739699 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:22.740270 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:22.740288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:22.741332 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:22.741336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:22.742576 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:22.742590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:22.743364 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:22.743371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:22.743645 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:22.744406 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:22.744436 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:22.746585 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:22.749677 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:22.749688 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:22.749703 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:22.750153 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:22.750552 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:22.750558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:22.750561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:22.750748 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:22.751281 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:22.751290 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:22.751695 (+ 405us) client_negotiation.cc:769] Sending connection context
0812 16:38:22.751966 (+ 271us) client_negotiation.cc:240] Negotiation successful
0812 16:38:22.752516 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":265,"threads_started":1}
I20260812 16:38:22.755518 2199 heartbeater.cc:347] Connected to a master server at 127.31.61.190:39315
I20260812 16:38:22.755896 2199 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
I20260812 16:38:22.756788 2199 heartbeater.cc:511] Master 127.31.61.190:39315 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
I20260812 16:38:22.759387 2008 ts_manager.cc:194] Registered new tserver with Master: 30e45d8000b8406bb608abf2b51687d9 (127.31.61.129:34191)
I20260812 16:38:22.760973 2008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:40339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:38:22.783371 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39315
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:23.109215 2205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:23.109503 2205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:23.109624 2205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:23.116941 2205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:23.117062 2205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:23.117213 2205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:23.127213 2205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:23.128604 2205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:23.129904 2205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:23.143512 2210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:23.143582 2211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:23.143476 2213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:23.144197 2205 server_base.cc:1034] running on GCE node
I20260812 16:38:23.145042 2205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:23.146500 2205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:23.147737 2205 hybrid_clock.cc:648] HybridClock initialized: now 1786552703147675 us; error 47 us; skew 500 ppm
I20260812 16:38:23.148269 2205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:23 dist-test-slave-f7th krb5kdc[1962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552703, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:23.151957 2205 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:38:23.153926 2205 webserver.cc:460] Webserver started at http://127.31.61.130:39653/ using document root <none> and password file <none>
I20260812 16:38:23.154676 2205 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:23.154788 2205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:23.155113 2205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:23.158205 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "49d80ffc8e894315accfc1b735003546"
format_stamp: "Formatted at 2026-08-12 16:38:23 on dist-test-slave-f7th"
server_key: "e1b521a75e79fabcf467e3783c3441b5"
server_key_iv: "b24314849f81c326fad36bf4c568c307"
server_key_version: "encryptionkey@0"
I20260812 16:38:23.159010 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "49d80ffc8e894315accfc1b735003546"
format_stamp: "Formatted at 2026-08-12 16:38:23 on dist-test-slave-f7th"
server_key: "e1b521a75e79fabcf467e3783c3441b5"
server_key_iv: "b24314849f81c326fad36bf4c568c307"
server_key_version: "encryptionkey@0"
I20260812 16:38:23.164718 2205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260812 16:38:23.168421 2220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:23.169891 2205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:23.170056 2205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "49d80ffc8e894315accfc1b735003546"
format_stamp: "Formatted at 2026-08-12 16:38:23 on dist-test-slave-f7th"
server_key: "e1b521a75e79fabcf467e3783c3441b5"
server_key_iv: "b24314849f81c326fad36bf4c568c307"
server_key_version: "encryptionkey@0"
I20260812 16:38:23.170253 2205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:23.189260 2205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:23.193444 2205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:23.193723 2205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:23.194705 2205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:23.196107 2205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:23.196178 2205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:23.196306 2205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:23.196353 2205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:23.227213 2205 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:33935
I20260812 16:38:23.227249 2333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:33935 every 8 connection(s)
I20260812 16:38:23.228529 2205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:23.232865 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 2205
I20260812 16:38:23.233075 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:23.233639 31990 external_mini_cluster.cc:1428] Setting key cb9f0b8d7453d096de4dc952161e6b9f
Aug 12 16:38:23 dist-test-slave-f7th krb5kdc[1962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552703, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:23.251736 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:23.231229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:35043 (local address 127.31.61.190:39315)
0812 16:38:23.231398 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:23.231403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:23.232608 (+ 1205us) server_negotiation.cc:407] Connection header received
0812 16:38:23.235042 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:23.235046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:23.235121 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:23.235207 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:23.237171 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:23.238061 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:23.240222 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:23.240816 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:23.240917 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:23.245222 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:23.245229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:23.245233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:23.245273 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:23.248131 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:23.249071 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:23.249077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:23.249079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:23.249254 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:23.249780 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:23.249786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:23.249790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:23.250108 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:23.250258 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:23.251051 (+ 793us) server_negotiation.cc:299] Negotiation successful
0812 16:38:23.251304 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
I20260812 16:38:23.252853 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:23.231633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:39315 (local address 127.31.61.130:35043)
0812 16:38:23.232346 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:23.232390 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:23.234699 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:23.235492 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:23.235506 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:23.236179 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:23.236983 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:23.237009 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:23.238229 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:23.238233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:23.240050 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:23.240067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:23.241272 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:23.241278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:23.241560 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:23.242277 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:23.242307 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:23.244932 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:23.248301 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:23.248313 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:23.248334 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:23.248923 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:23.249378 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:23.249384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:23.249388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:23.249653 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:23.250247 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:23.250257 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:23.250648 (+ 391us) client_negotiation.cc:769] Sending connection context
0812 16:38:23.250921 (+ 273us) client_negotiation.cc:240] Negotiation successful
0812 16:38:23.251505 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":270,"threads_started":1}
I20260812 16:38:23.254606 2334 heartbeater.cc:347] Connected to a master server at 127.31.61.190:39315
I20260812 16:38:23.254906 2334 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
I20260812 16:38:23.255757 2334 heartbeater.cc:511] Master 127.31.61.190:39315 requested a full tablet report, sending...
I20260812 16:38:23.257575 2008 ts_manager.cc:194] Registered new tserver with Master: 49d80ffc8e894315accfc1b735003546 (127.31.61.130:33935)
I20260812 16:38:23.258692 2008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:35043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:38:23.279842 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39315
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:23.608259 2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:23.608564 2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:23.608664 2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:23.616181 2339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:23.616313 2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:23.616426 2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:23.626788 2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:23.628194 2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:23.629487 2339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:23.642168 2347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:23.642302 2344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:23.642334 2345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:23.643167 2339 server_base.cc:1034] running on GCE node
I20260812 16:38:23.643798 2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:23.644690 2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:23.645931 2339 hybrid_clock.cc:648] HybridClock initialized: now 1786552703645876 us; error 46 us; skew 500 ppm
I20260812 16:38:23.646257 2339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:23 dist-test-slave-f7th krb5kdc[1962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552703, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:23.649392 2339 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:38:23.650883 2339 webserver.cc:460] Webserver started at http://127.31.61.131:34081/ using document root <none> and password file <none>
I20260812 16:38:23.651566 2339 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:23.651722 2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:23.651993 2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:23.654894 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "f6de97f737fe4cf7bea1d2745ef37d47"
format_stamp: "Formatted at 2026-08-12 16:38:23 on dist-test-slave-f7th"
server_key: "5d14d9c2a914c5e62c860299ebd8c630"
server_key_iv: "84c39c1e2ce87451263afb51024d0581"
server_key_version: "encryptionkey@0"
I20260812 16:38:23.655735 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "f6de97f737fe4cf7bea1d2745ef37d47"
format_stamp: "Formatted at 2026-08-12 16:38:23 on dist-test-slave-f7th"
server_key: "5d14d9c2a914c5e62c860299ebd8c630"
server_key_iv: "84c39c1e2ce87451263afb51024d0581"
server_key_version: "encryptionkey@0"
I20260812 16:38:23.660920 2339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:38:23.664361 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:23.665676 2339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:23.665824 2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "f6de97f737fe4cf7bea1d2745ef37d47"
format_stamp: "Formatted at 2026-08-12 16:38:23 on dist-test-slave-f7th"
server_key: "5d14d9c2a914c5e62c860299ebd8c630"
server_key_iv: "84c39c1e2ce87451263afb51024d0581"
server_key_version: "encryptionkey@0"
I20260812 16:38:23.666009 2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:23.683663 2339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:23.687768 2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:23.688026 2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:23.688964 2339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:23.690414 2339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:23.690496 2339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:23.690620 2339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:23.690678 2339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:23.723034 2339 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:35689
I20260812 16:38:23.723084 2468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:35689 every 8 connection(s)
I20260812 16:38:23.724304 2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:23.729738 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 2339
I20260812 16:38:23.729962 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:23.730525 31990 external_mini_cluster.cc:1428] Setting key 773ef3e8833eefcc06ac28b3c1f2ec1a
Aug 12 16:38:23 dist-test-slave-f7th krb5kdc[1962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552703, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:23.745654 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:23.726878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39959 (local address 127.31.61.190:39315)
0812 16:38:23.727075 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:23.727080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:23.728287 (+ 1207us) server_negotiation.cc:407] Connection header received
0812 16:38:23.730089 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:23.730097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:23.730231 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:23.730375 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:23.732096 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:23.733319 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:23.735074 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:23.735487 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:23.735561 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:23.739319 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:23.739324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:23.739329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:23.739385 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:23.742003 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:23.742928 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:23.742932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:23.742940 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:23.743097 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:23.743654 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:23.743660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:23.743664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:23.744027 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:23.744203 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:23.745051 (+ 848us) server_negotiation.cc:299] Negotiation successful
0812 16:38:23.745307 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:38:23.746769 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:23.727294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:39315 (local address 127.31.61.131:39959)
0812 16:38:23.728117 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:23.728155 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:23.729786 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:23.730557 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:23.730565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:23.731151 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:23.731920 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:23.731948 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:23.733528 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:23.733534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:23.734882 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:23.734895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:23.735834 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:23.735840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:23.736047 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:23.736803 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:23.736826 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:23.739086 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:23.742149 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:23.742157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:23.742170 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:23.742704 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:23.743233 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:23.743237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:23.743239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:23.743485 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:23.744186 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:23.744193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:23.744617 (+ 424us) client_negotiation.cc:769] Sending connection context
0812 16:38:23.744909 (+ 292us) client_negotiation.cc:240] Negotiation successful
0812 16:38:23.745455 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":405,"threads_started":1}
I20260812 16:38:23.748240 2469 heartbeater.cc:347] Connected to a master server at 127.31.61.190:39315
I20260812 16:38:23.748610 2469 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:23.749414 2469 heartbeater.cc:511] Master 127.31.61.190:39315 requested a full tablet report, sending...
I20260812 16:38:23.750946 2008 ts_manager.cc:194] Registered new tserver with Master: f6de97f737fe4cf7bea1d2745ef37d47 (127.31.61.131:35689)
I20260812 16:38:23.751808 2008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:39959
I20260812 16:38:23.759298 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:38:23.764194 2199 heartbeater.cc:503] Master 127.31.61.190:39315 was elected leader, sending a full tablet report...
W20260812 16:38:23.790421 2201 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:38:23.787838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.31.61.190:39315)
0812 16:38:23.788052 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:23.788057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:23.788443 (+ 386us) server_negotiation.cc:407] Connection header received
0812 16:38:23.788594 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:23.788599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:23.788666 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:23.788762 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:23.790141 (+ 1379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48466: BlockingRecv error: recv got EOF from 127.0.0.1:48466 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 16:38:23.793642 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2069
W20260812 16:38:23.796309 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2069: Not authorized: Client connection negotiation failed: client connection to 127.31.61.129:34191: server requires authentication, but client does not have Kerberos credentials available
I20260812 16:38:23.796519 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 2069
I20260812 16:38:23.810912 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2205
W20260812 16:38:23.813480 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2205: Not authorized: Client connection negotiation failed: client connection to 127.31.61.130:33935: server requires authentication, but client does not have Kerberos credentials available
I20260812 16:38:23.813681 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 2205
I20260812 16:38:23.827464 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2339
W20260812 16:38:23.830663 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2339: Not authorized: Client connection negotiation failed: client connection to 127.31.61.131:35689: server requires authentication, but client does not have Kerberos credentials available
I20260812 16:38:23.830828 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 2339
I20260812 16:38:23.845412 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 1976
I20260812 16:38:23.846768 2038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:23.950947 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 1976
2026-08-12T16:38:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3497 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:24 dist-test-slave-f7th krb5kdc[2486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:24 dist-test-slave-f7th krb5kdc[2486](info): set up 2 sockets
Aug 12 16:38:24 dist-test-slave-f7th krb5kdc[2486](info): commencing operation
krb5kdc: starting...
W20260812 16:38:24.983953 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:25 dist-test-slave-f7th krb5kdc[2486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:38:25.208683 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:33185
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:33185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:25.552484 2499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:25.552807 2499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:25.552943 2499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:25.560359 2499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:25.560503 2499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:25.560585 2499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:25.560660 2499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:25.560729 2499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:25.570848 2499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:33185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:33185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:25.572407 2499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:25.573817 2499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:25.585871 2504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:25.585871 2505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:25.585816 2507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:25.586798 2499 server_base.cc:1034] running on GCE node
I20260812 16:38:25.587584 2499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:25.589109 2499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:25.590337 2499 hybrid_clock.cc:648] HybridClock initialized: now 1786552705590283 us; error 48 us; skew 500 ppm
I20260812 16:38:25.590829 2499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:25 dist-test-slave-f7th krb5kdc[2486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552705, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:25.600687 2499 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:38:25.602324 2499 webserver.cc:460] Webserver started at http://127.31.61.190:40107/ using document root <none> and password file <none>
I20260812 16:38:25.603050 2499 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:25.603175 2499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:25.603487 2499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:25.606905 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "81e66adc8acc4c25a5622faa489660ab"
format_stamp: "Formatted at 2026-08-12 16:38:25 on dist-test-slave-f7th"
server_key: "aba1568866270d8624914200bb297994"
server_key_iv: "1e1760f68256693cc6c75b47dc25b43f"
server_key_version: "encryptionkey@0"
I20260812 16:38:25.607869 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "81e66adc8acc4c25a5622faa489660ab"
format_stamp: "Formatted at 2026-08-12 16:38:25 on dist-test-slave-f7th"
server_key: "aba1568866270d8624914200bb297994"
server_key_iv: "1e1760f68256693cc6c75b47dc25b43f"
server_key_version: "encryptionkey@0"
I20260812 16:38:25.613606 2499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:38:25.617444 2514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:25.619012 2499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:38:25.619191 2499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "81e66adc8acc4c25a5622faa489660ab"
format_stamp: "Formatted at 2026-08-12 16:38:25 on dist-test-slave-f7th"
server_key: "aba1568866270d8624914200bb297994"
server_key_iv: "1e1760f68256693cc6c75b47dc25b43f"
server_key_version: "encryptionkey@0"
I20260812 16:38:25.619469 2499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:25.640468 2499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:25.645867 2499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:25.646162 2499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:25.665680 2499 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:33185
I20260812 16:38:25.665726 2565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:33185 every 8 connection(s)
I20260812 16:38:25.667054 2499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:25.670632 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 2499
I20260812 16:38:25.670856 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:25.671165 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:25.671690 31990 external_mini_cluster.cc:1428] Setting key 818b7ca24c0d27ac0ebb682a910353be
I20260812 16:38:25.684878 2566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab: Bootstrap starting.
Aug 12 16:38:25 dist-test-slave-f7th krb5kdc[2486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:25.688531 2566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:25.689872 2566 log.cc:824] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:25.693184 2566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab: No bootstrap required, opened a new log
I20260812 16:38:25.695271 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:25.674299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.31.61.190:33185)
0812 16:38:25.675192 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:25.675204 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:25.675250 (+ 46us) server_negotiation.cc:407] Connection header received
0812 16:38:25.677032 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:25.677056 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:25.677468 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:25.677853 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:25.679321 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:25.680624 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:25.681874 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:25.682617 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:25.682789 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:25.687015 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:25.687023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:25.687037 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:25.687080 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:25.690620 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:25.691387 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:25.691396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:25.691406 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:25.691635 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:25.692001 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:25.692006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:25.692009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:25.692737 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:25.693014 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:25.693464 (+ 450us) server_negotiation.cc:299] Negotiation successful
0812 16:38:25.693915 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":372,"threads_started":1}
I20260812 16:38:25.699537 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:38:25.703377 2566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66adc8acc4c25a5622faa489660ab" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33185 } }
I20260812 16:38:25.703882 2566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:25.703977 2566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e66adc8acc4c25a5622faa489660ab, State: Initialized, Role: FOLLOWER
I20260812 16:38:25.704646 2566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66adc8acc4c25a5622faa489660ab" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33185 } }
I20260812 16:38:25.704916 2566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:25.705044 2566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:25.705202 2566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:25.708233 2566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66adc8acc4c25a5622faa489660ab" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33185 } }
I20260812 16:38:25.708783 2566 leader_election.cc:302] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e66adc8acc4c25a5622faa489660ab; no voters:
I20260812 16:38:25.709364 2566 leader_election.cc:288] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:25.709522 2572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:25.710116 2572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [term 1 LEADER]: Becoming Leader. State: Replica: 81e66adc8acc4c25a5622faa489660ab, State: Running, Role: LEADER
I20260812 16:38:25.710752 2572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66adc8acc4c25a5622faa489660ab" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33185 } }
I20260812 16:38:25.713593 2566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:25.713452 2574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e66adc8acc4c25a5622faa489660ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66adc8acc4c25a5622faa489660ab" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33185 } } }
I20260812 16:38:25.713894 2574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:25.714355 2573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e66adc8acc4c25a5622faa489660ab. Latest consensus state: current_term: 1 leader_uuid: "81e66adc8acc4c25a5622faa489660ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66adc8acc4c25a5622faa489660ab" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33185 } } }
I20260812 16:38:25.714618 2573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:25.724994 2576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:25.730396 2576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:25.740736 2576 catalog_manager.cc:1347] Generated new cluster ID: 154cdb7062bd4a4ab190058d19bf2176
I20260812 16:38:25.740849 2576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:25.782115 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:25.768271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.31.61.190:33185)
0812 16:38:25.768441 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:25.768451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:25.772801 (+ 4350us) server_negotiation.cc:407] Connection header received
0812 16:38:25.773264 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:25.773270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:25.773351 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:25.773422 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:25.774746 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:25.775874 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:25.778042 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:25.778693 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:25.778787 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:25.780266 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:25.780273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:25.780278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:25.781244 (+ 966us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:25.781318 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:25.781348 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:25.781430 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:25.781656 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 16:38:25.781758 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 16:38:25.791404 2576 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:25.792743 2576 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:25.798599 2576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e66adc8acc4c25a5622faa489660ab: Generated new TSK 0
I20260812 16:38:25.799731 2576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:38:25.811180 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2499
I20260812 16:38:25.813218 2561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:25.905258 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 2499
2026-08-12T16:38:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1948 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:25 dist-test-slave-f7th krb5kdc[2601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:25 dist-test-slave-f7th krb5kdc[2601](info): set up 2 sockets
Aug 12 16:38:25 dist-test-slave-f7th krb5kdc[2601](info): commencing operation
krb5kdc: starting...
W20260812 16:38:26.931452 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:27 dist-test-slave-f7th krb5kdc[2601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:38:27.133710 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:40569
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:40569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:27.465372 2614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:27.465678 2614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:27.465794 2614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:27.473169 2614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:27.473292 2614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:27.473340 2614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:27.473419 2614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:27.473470 2614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:27.483523 2614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:40569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:40569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:27.484908 2614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:27.486217 2614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:27.497368 2619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:27.497514 2622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:27.497658 2620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:27.498487 2614 server_base.cc:1034] running on GCE node
I20260812 16:38:27.499264 2614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:27.500725 2614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:27.501945 2614 hybrid_clock.cc:648] HybridClock initialized: now 1786552707501903 us; error 32 us; skew 500 ppm
I20260812 16:38:27.502310 2614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:27 dist-test-slave-f7th krb5kdc[2601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552707, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:27.505722 2614 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:38:27.507393 2614 webserver.cc:460] Webserver started at http://127.31.61.190:32941/ using document root <none> and password file <none>
I20260812 16:38:27.508155 2614 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:27.508267 2614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:27.508574 2614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:27.511655 2614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "9e393d849e7145b9a0dc95735c1c0bdf"
format_stamp: "Formatted at 2026-08-12 16:38:27 on dist-test-slave-f7th"
server_key: "e117bcba9982dd21c868879bb5198a52"
server_key_iv: "a19a7df4da1b5807aa2943946a4ea6dc"
server_key_version: "encryptionkey@0"
I20260812 16:38:27.512420 2614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "9e393d849e7145b9a0dc95735c1c0bdf"
format_stamp: "Formatted at 2026-08-12 16:38:27 on dist-test-slave-f7th"
server_key: "e117bcba9982dd21c868879bb5198a52"
server_key_iv: "a19a7df4da1b5807aa2943946a4ea6dc"
server_key_version: "encryptionkey@0"
I20260812 16:38:27.517769 2614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:38:27.521732 2629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:27.523159 2614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:38:27.523337 2614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "9e393d849e7145b9a0dc95735c1c0bdf"
format_stamp: "Formatted at 2026-08-12 16:38:27 on dist-test-slave-f7th"
server_key: "e117bcba9982dd21c868879bb5198a52"
server_key_iv: "a19a7df4da1b5807aa2943946a4ea6dc"
server_key_version: "encryptionkey@0"
I20260812 16:38:27.523533 2614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:27.542541 2614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:27.546945 2614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:27.547246 2614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:27.565088 2614 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:40569
I20260812 16:38:27.565112 2680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:40569 every 8 connection(s)
I20260812 16:38:27.566414 2614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:27.570148 2681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:27.574295 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 2614
I20260812 16:38:27.574512 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:27.575139 31990 external_mini_cluster.cc:1428] Setting key cb3d9690b3a8f70be242adb19f33a078
I20260812 16:38:27.581645 2681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf: Bootstrap starting.
I20260812 16:38:27.585353 2681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:27.586525 2681 log.cc:824] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:27.589917 2681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf: No bootstrap required, opened a new log
Aug 12 16:38:27 dist-test-slave-f7th krb5kdc[2601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:27.598093 2681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e393d849e7145b9a0dc95735c1c0bdf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40569 } }
I20260812 16:38:27.598485 2681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:27.598603 2681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e393d849e7145b9a0dc95735c1c0bdf, State: Initialized, Role: FOLLOWER
I20260812 16:38:27.599287 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:27.577159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.31.61.190:40569)
0812 16:38:27.577942 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:27.577954 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:27.577997 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:38:27.580073 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:27.580100 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:27.580507 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:27.580845 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:27.582546 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:27.584029 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:27.585802 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:27.586783 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:27.586936 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:27.591482 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:27.591491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:27.591507 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:27.591552 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:27.595040 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:27.595803 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:27.595810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:27.595817 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:27.595981 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:27.596379 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:27.596383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:27.596385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:27.596992 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:27.597295 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:27.597716 (+ 421us) server_negotiation.cc:299] Negotiation successful
0812 16:38:27.598141 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":336,"threads_started":1}
I20260812 16:38:27.599243 2681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e393d849e7145b9a0dc95735c1c0bdf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40569 } }
I20260812 16:38:27.599473 2681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:27.599680 2681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:27.599856 2681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:27.602867 2681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e393d849e7145b9a0dc95735c1c0bdf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40569 } }
I20260812 16:38:27.603343 2681 leader_election.cc:302] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e393d849e7145b9a0dc95735c1c0bdf; no voters:
I20260812 16:38:27.603974 2681 leader_election.cc:288] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:27.604118 2686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:27.604772 2686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [term 1 LEADER]: Becoming Leader. State: Replica: 9e393d849e7145b9a0dc95735c1c0bdf, State: Running, Role: LEADER
I20260812 16:38:27.605288 2686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e393d849e7145b9a0dc95735c1c0bdf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40569 } }
I20260812 16:38:27.605964 2681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:27.608379 2688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e393d849e7145b9a0dc95735c1c0bdf. Latest consensus state: current_term: 1 leader_uuid: "9e393d849e7145b9a0dc95735c1c0bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e393d849e7145b9a0dc95735c1c0bdf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40569 } } }
I20260812 16:38:27.608397 2687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e393d849e7145b9a0dc95735c1c0bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e393d849e7145b9a0dc95735c1c0bdf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40569 } } }
I20260812 16:38:27.609087 2688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:27.609107 2687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:27.611925 2693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:27.616341 2693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:27.626735 2693 catalog_manager.cc:1347] Generated new cluster ID: f81c412aac8147529241c81507d24d11
I20260812 16:38:27.626924 2693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:27.654798 2693 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:27.656158 2693 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:27.661904 2693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e393d849e7145b9a0dc95735c1c0bdf: Generated new TSK 0
I20260812 16:38:27.662926 2693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:38:27.729082 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:40569
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:28.085064 2707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:28.085382 2707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:28.085530 2707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:28.092864 2707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:28.093019 2707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:28.093206 2707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:28.103538 2707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:40569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:28.104890 2707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:28.106195 2707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:28.120031 2712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:28.120013 2715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:28.120059 2713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:28.121134 2707 server_base.cc:1034] running on GCE node
I20260812 16:38:28.121794 2707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:28.122795 2707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:28.124004 2707 hybrid_clock.cc:648] HybridClock initialized: now 1786552708123958 us; error 36 us; skew 500 ppm
I20260812 16:38:28.124317 2707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:28 dist-test-slave-f7th krb5kdc[2601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552708, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:28.127880 2707 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:38:28.129501 2707 webserver.cc:460] Webserver started at http://127.31.61.129:41165/ using document root <none> and password file <none>
I20260812 16:38:28.130460 2707 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:28.130579 2707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:28.130908 2707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:28.134119 2707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "cb639a70e8024244bb8786bb6a465eb3"
format_stamp: "Formatted at 2026-08-12 16:38:28 on dist-test-slave-f7th"
server_key: "3bdde71087ecce4eb4af543893c350ec"
server_key_iv: "6af695c87147eeff0b6a7098c4f7bcd5"
server_key_version: "encryptionkey@0"
I20260812 16:38:28.134934 2707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "cb639a70e8024244bb8786bb6a465eb3"
format_stamp: "Formatted at 2026-08-12 16:38:28 on dist-test-slave-f7th"
server_key: "3bdde71087ecce4eb4af543893c350ec"
server_key_iv: "6af695c87147eeff0b6a7098c4f7bcd5"
server_key_version: "encryptionkey@0"
I20260812 16:38:28.140655 2707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:38:28.144898 2722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:28.146529 2707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:38:28.146704 2707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "cb639a70e8024244bb8786bb6a465eb3"
format_stamp: "Formatted at 2026-08-12 16:38:28 on dist-test-slave-f7th"
server_key: "3bdde71087ecce4eb4af543893c350ec"
server_key_iv: "6af695c87147eeff0b6a7098c4f7bcd5"
server_key_version: "encryptionkey@0"
I20260812 16:38:28.146903 2707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:28.177789 2707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:28.182447 2707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:28.182739 2707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:28.183825 2707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:28.185225 2707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:28.185298 2707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:28.185413 2707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:28.185465 2707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:28.217658 2707 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:43373
I20260812 16:38:28.217696 2834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:43373 every 8 connection(s)
I20260812 16:38:28.218943 2707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:28.222886 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 2707
I20260812 16:38:28.223131 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:28.223937 31990 external_mini_cluster.cc:1428] Setting key 11f7cd3aadc6e4649e857e12b9e97ac6
Aug 12 16:38:28 dist-test-slave-f7th krb5kdc[2601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552708, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:28.241319 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:28.221517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:57205 (local address 127.31.61.190:40569)
0812 16:38:28.222005 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:28.222010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:28.222982 (+ 972us) server_negotiation.cc:407] Connection header received
0812 16:38:28.225267 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:28.225273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:28.225352 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:28.225448 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:28.227474 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:28.228370 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:28.230450 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:28.230876 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:28.230987 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:28.234969 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:28.234975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:28.234978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:28.235028 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:28.237645 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:28.238654 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:28.238660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:28.238663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:28.238818 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:28.239405 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:28.239410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:28.239414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:28.239770 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:28.239905 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:28.240852 (+ 947us) server_negotiation.cc:299] Negotiation successful
0812 16:38:28.241037 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
I20260812 16:38:28.242861 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:28.221942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:40569 (local address 127.31.61.129:57205)
0812 16:38:28.222785 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:28.222843 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:28.224989 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:28.225700 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:28.225708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:28.226404 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:28.227253 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:28.227281 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:28.228525 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:28.228530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:28.230292 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:28.230308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:28.231106 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:28.231112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:28.231379 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:28.232108 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:28.232131 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:28.234685 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:28.237798 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:28.237807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:28.237826 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:28.238499 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:28.238954 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:28.238958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:28.238962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:28.239219 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:28.239905 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:28.239915 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:28.240400 (+ 485us) client_negotiation.cc:769] Sending connection context
0812 16:38:28.240754 (+ 354us) client_negotiation.cc:240] Negotiation successful
0812 16:38:28.241371 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":290,"threads_started":1}
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
I20260812 16:38:28.244510 2835 heartbeater.cc:347] Connected to a master server at 127.31.61.190:40569
I20260812 16:38:28.244834 2835 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:28.245708 2835 heartbeater.cc:511] Master 127.31.61.190:40569 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
I20260812 16:38:28.248601 2646 ts_manager.cc:194] Registered new tserver with Master: cb639a70e8024244bb8786bb6a465eb3 (127.31.61.129:43373)
I20260812 16:38:28.250519 2646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:57205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:38:28.275761 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:40569
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:28.609771 2841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:28.610098 2841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:28.610255 2841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:28.617743 2841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:28.617892 2841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:28.618064 2841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:28.628309 2841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:40569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:28.629683 2841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:28.631049 2841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:28.644474 2847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:28.644451 2849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:28.644789 2846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:28.645552 2841 server_base.cc:1034] running on GCE node
I20260812 16:38:28.646225 2841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:28.647225 2841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:28.648453 2841 hybrid_clock.cc:648] HybridClock initialized: now 1786552708648400 us; error 42 us; skew 500 ppm
I20260812 16:38:28.648803 2841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:28 dist-test-slave-f7th krb5kdc[2601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552708, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:28.651909 2841 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:38:28.653368 2841 webserver.cc:460] Webserver started at http://127.31.61.130:39539/ using document root <none> and password file <none>
I20260812 16:38:28.654072 2841 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:28.654187 2841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:28.654479 2841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:28.657363 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "46e1457ed874462795f9774855774c0f"
format_stamp: "Formatted at 2026-08-12 16:38:28 on dist-test-slave-f7th"
server_key: "a2e6a078b038e8cc23f029c15e2bb067"
server_key_iv: "190fcd13a9e9d1a32b67760d763c0b6f"
server_key_version: "encryptionkey@0"
I20260812 16:38:28.658136 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "46e1457ed874462795f9774855774c0f"
format_stamp: "Formatted at 2026-08-12 16:38:28 on dist-test-slave-f7th"
server_key: "a2e6a078b038e8cc23f029c15e2bb067"
server_key_iv: "190fcd13a9e9d1a32b67760d763c0b6f"
server_key_version: "encryptionkey@0"
I20260812 16:38:28.663389 2841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:38:28.666944 2856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:28.668334 2841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:28.668495 2841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "46e1457ed874462795f9774855774c0f"
format_stamp: "Formatted at 2026-08-12 16:38:28 on dist-test-slave-f7th"
server_key: "a2e6a078b038e8cc23f029c15e2bb067"
server_key_iv: "190fcd13a9e9d1a32b67760d763c0b6f"
server_key_version: "encryptionkey@0"
I20260812 16:38:28.668695 2841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:28.687830 2841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:28.692021 2841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:28.692312 2841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:28.693291 2841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:28.694769 2841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:28.694849 2841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:28.694972 2841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:28.695024 2841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:28.727254 2841 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:38167
I20260812 16:38:28.727299 2968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:38167 every 8 connection(s)
I20260812 16:38:28.728684 2841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:28.735770 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 2841
I20260812 16:38:28.735935 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:28.736421 31990 external_mini_cluster.cc:1428] Setting key 88cc8a529a12c2e609da03eb74019a4d
Aug 12 16:38:28 dist-test-slave-f7th krb5kdc[2601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552708, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:28.749810 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:28.731294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:44443 (local address 127.31.61.190:40569)
0812 16:38:28.731516 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:28.731521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:28.732754 (+ 1233us) server_negotiation.cc:407] Connection header received
0812 16:38:28.734508 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:28.734513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:28.734584 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:28.734670 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:28.736238 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:28.737245 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:28.738956 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:28.739580 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:28.739737 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:28.743172 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:28.743179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:28.743183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:28.743220 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:28.746175 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:28.747176 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:28.747181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:28.747184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:28.747322 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:28.747896 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:28.747900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:28.747903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:28.748205 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:28.748335 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:28.749327 (+ 992us) server_negotiation.cc:299] Negotiation successful
0812 16:38:28.749536 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 16:38:28.751502 2971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:28.731841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:40569 (local address 127.31.61.130:44443)
0812 16:38:28.732582 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:28.732631 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:28.734269 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:28.734857 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:28.734865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:28.735382 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:28.736064 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:28.736085 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:28.737401 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:28.737407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:28.738802 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:28.738815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:28.739838 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:28.739843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:28.740054 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:28.740704 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:28.740732 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:28.742930 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:28.746368 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:28.746381 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:28.746400 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:28.747027 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:28.747437 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:28.747443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:28.747446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:28.747746 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:28.748363 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:28.748373 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:28.748865 (+ 492us) client_negotiation.cc:769] Sending connection context
0812 16:38:28.749205 (+ 340us) client_negotiation.cc:240] Negotiation successful
0812 16:38:28.749806 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":288,"threads_started":1}
I20260812 16:38:28.753260 2969 heartbeater.cc:347] Connected to a master server at 127.31.61.190:40569
I20260812 16:38:28.753628 2969 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:28.754536 2969 heartbeater.cc:511] Master 127.31.61.190:40569 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
I20260812 16:38:28.756299 2646 ts_manager.cc:194] Registered new tserver with Master: 46e1457ed874462795f9774855774c0f (127.31.61.130:38167)
I20260812 16:38:28.757202 2646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:44443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:38:28.785079 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:40569
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:29.131049 2974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:29.131376 2974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:29.131517 2974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:29.139434 2974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:29.139580 2974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:29.139752 2974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:29.150771 2974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:40569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:29.152549 2974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:29.154201 2974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:29.168260 2979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:29.168267 2982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:29.168274 2980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:29.169009 2974 server_base.cc:1034] running on GCE node
I20260812 16:38:29.169669 2974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:29.170703 2974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:29.171947 2974 hybrid_clock.cc:648] HybridClock initialized: now 1786552709171900 us; error 33 us; skew 500 ppm
I20260812 16:38:29.172389 2974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[2601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552709, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:29.176077 2974 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:38:29.177641 2974 webserver.cc:460] Webserver started at http://127.31.61.131:46499/ using document root <none> and password file <none>
I20260812 16:38:29.178318 2974 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:29.178447 2974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:29.178740 2974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:29.181723 2974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "a30fb118f26e49fe9462333a34cbcd7e"
format_stamp: "Formatted at 2026-08-12 16:38:29 on dist-test-slave-f7th"
server_key: "7e793d0554d9784c96950d1b42578b52"
server_key_iv: "9d17f14967866cfcca79aa4a2d76a864"
server_key_version: "encryptionkey@0"
I20260812 16:38:29.182507 2974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "a30fb118f26e49fe9462333a34cbcd7e"
format_stamp: "Formatted at 2026-08-12 16:38:29 on dist-test-slave-f7th"
server_key: "7e793d0554d9784c96950d1b42578b52"
server_key_iv: "9d17f14967866cfcca79aa4a2d76a864"
server_key_version: "encryptionkey@0"
I20260812 16:38:29.187906 2974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 16:38:29.191838 2989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:29.193320 2974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:29.193488 2974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "a30fb118f26e49fe9462333a34cbcd7e"
format_stamp: "Formatted at 2026-08-12 16:38:29 on dist-test-slave-f7th"
server_key: "7e793d0554d9784c96950d1b42578b52"
server_key_iv: "9d17f14967866cfcca79aa4a2d76a864"
server_key_version: "encryptionkey@0"
I20260812 16:38:29.193687 2974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:29.214289 2974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:29.219460 2974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:29.219822 2974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:29.221066 2974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:29.222851 2974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:29.222962 2974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:29.223089 2974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:29.223150 2974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:29.254228 2835 heartbeater.cc:503] Master 127.31.61.190:40569 was elected leader, sending a full tablet report...
I20260812 16:38:29.257225 2974 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:35995
I20260812 16:38:29.257270 3101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:35995 every 8 connection(s)
I20260812 16:38:29.258533 2974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:29.265981 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 2974
I20260812 16:38:29.266130 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:29.266537 31990 external_mini_cluster.cc:1428] Setting key 5453172f7ef35266bcbf2731687da178
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[2601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552709, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:29.278316 3104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:29.260884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:52947 (local address 127.31.61.190:40569)
0812 16:38:29.261404 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:29.261410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:29.262327 (+ 917us) server_negotiation.cc:407] Connection header received
0812 16:38:29.264098 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:29.264103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:29.264170 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:29.264264 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:29.265604 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.266449 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.268259 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.268692 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.268804 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:29.272372 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:29.272378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:29.272382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:29.272418 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:29.274938 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.275730 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.275736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.275738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.275899 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.276391 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.276397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.276401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.276709 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:29.276868 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:29.277634 (+ 766us) server_negotiation.cc:299] Negotiation successful
0812 16:38:29.277884 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
I20260812 16:38:29.279263 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:29.261475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:40569 (local address 127.31.61.131:52947)
0812 16:38:29.262177 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:29.262219 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:29.263842 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:29.264440 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:29.264448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:29.264944 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:29.265472 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:29.265489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.266618 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.266623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:29.268124 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:29.268137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.268914 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.268921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:29.269137 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:29.269723 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:29.269750 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:29.272127 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:29.275088 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:29.275098 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:29.275117 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:29.275581 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:29.276017 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:29.276023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:29.276027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:29.276256 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:29.276843 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:29.276852 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:29.277211 (+ 359us) client_negotiation.cc:769] Sending connection context
0812 16:38:29.277483 (+ 272us) client_negotiation.cc:240] Negotiation successful
0812 16:38:29.278032 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":262,"threads_started":1}
I20260812 16:38:29.280726 3102 heartbeater.cc:347] Connected to a master server at 127.31.61.190:40569
I20260812 16:38:29.281041 3102 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:29.281797 3102 heartbeater.cc:511] Master 127.31.61.190:40569 requested a full tablet report, sending...
I20260812 16:38:29.283212 2646 ts_manager.cc:194] Registered new tserver with Master: a30fb118f26e49fe9462333a34cbcd7e (127.31.61.131:35995)
I20260812 16:38:29.284116 2646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:52947
I20260812 16:38:29.294239 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[2601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552709, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[2601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552709, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:29.345774 3104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:29.329565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.31.61.190:40569)
0812 16:38:29.329748 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:29.329753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:29.330239 (+ 486us) server_negotiation.cc:407] Connection header received
0812 16:38:29.330444 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:29.330449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:29.330529 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:29.330633 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:29.332379 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.333124 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.334392 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.334920 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.335043 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:29.339930 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:29.339938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:29.339944 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:29.339992 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:29.342956 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.343726 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.343733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.343737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.343876 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.344262 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.344266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.344268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.344631 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:29.344878 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:29.345231 (+ 353us) server_negotiation.cc:299] Negotiation successful
0812 16:38:29.345480 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 16:38:29.346367 2646 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36352
I20260812 16:38:29.348742 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2707
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[2601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552709, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:29.367094 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:29.349717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.31.61.129:43373)
0812 16:38:29.350298 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:29.350305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:29.350332 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:38:29.350400 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:29.350409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:29.350656 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:29.350797 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:29.352150 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.353495 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.354808 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.355398 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.355541 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:29.360361 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:29.360377 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:29.360394 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:29.360475 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:29.363296 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.364123 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.364132 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.364134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.364278 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.364943 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.364950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.364952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.365602 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:29.365928 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:29.366271 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 16:38:29.366608 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":299,"threads_started":1}
W20260812 16:38:29.368582 2830 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36418
W20260812 16:38:29.369827 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2707: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:38:29.370107 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 2707
I20260812 16:38:29.384250 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2841
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[2601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552709, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:38:29.399960 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:29.385143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.31.61.130:38167)
0812 16:38:29.385673 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:29.385681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:29.385706 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:38:29.385772 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:29.385779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:29.385983 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:29.386127 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:29.387346 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.388406 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.389608 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.390156 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.390275 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:29.394415 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:29.394428 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:29.394443 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:29.394484 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:29.397251 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.397899 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.397906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.397908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.398038 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.398468 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.398473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.398476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.398873 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:29.399108 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:29.399381 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 16:38:29.399651 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":240,"threads_started":1}
W20260812 16:38:29.401298 2964 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37070
W20260812 16:38:29.402417 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2841: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:38:29.402565 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 2841
I20260812 16:38:29.416399 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2974
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[2601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552709, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:38:29.433475 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:29.417325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.31.61.131:35995)
0812 16:38:29.417949 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:29.417956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:29.417995 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:38:29.418062 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:29.418069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:29.418272 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:29.418406 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:29.419623 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.420671 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.421912 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:29.422621 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:29.422736 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:29.427235 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:29.427251 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:29.427268 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:29.427308 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:29.430106 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.430772 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.430780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.430783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.430910 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:29.431300 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:29.431305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:29.431307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:29.431754 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:29.432029 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:29.432611 (+ 582us) server_negotiation.cc:299] Negotiation successful
0812 16:38:29.432963 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":281,"threads_started":1}
W20260812 16:38:29.434942 3097 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46274
W20260812 16:38:29.436187 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2974: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:38:29.436372 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 2974
I20260812 16:38:29.450563 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 2614
I20260812 16:38:29.451962 2676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:29.566274 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 2614
2026-08-12T16:38:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3666 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[3120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[3120](info): set up 2 sockets
Aug 12 16:38:29 dist-test-slave-f7th krb5kdc[3120](info): commencing operation
krb5kdc: starting...
W20260812 16:38:30.597404 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:30 dist-test-slave-f7th krb5kdc[3120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:38:30.799590 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:41757
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:41757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:31.126860 3133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:31.127166 3133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:31.127281 3133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:31.134661 3133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:31.134784 3133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:31.134833 3133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:31.134913 3133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:31.134967 3133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:31.145637 3133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:41757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:41757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:31.147047 3133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:31.148422 3133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:31.161101 3138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:31.161013 3141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:31.161039 3139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:31.161998 3133 server_base.cc:1034] running on GCE node
I20260812 16:38:31.162844 3133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:31.164498 3133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:31.165742 3133 hybrid_clock.cc:648] HybridClock initialized: now 1786552711165687 us; error 48 us; skew 500 ppm
I20260812 16:38:31.166139 3133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:31 dist-test-slave-f7th krb5kdc[3120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552711, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:31.175051 3133 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:38:31.176867 3133 webserver.cc:460] Webserver started at http://127.31.61.190:38379/ using document root <none> and password file <none>
I20260812 16:38:31.177649 3133 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:31.177788 3133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:31.178194 3133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:31.181609 3133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "d718d5598bf74e43ace4151e8792e42b"
format_stamp: "Formatted at 2026-08-12 16:38:31 on dist-test-slave-f7th"
server_key: "217c19279a3d623c16a77594d802d2eb"
server_key_iv: "831bdc3270f3cb7f31e4bdbdd58ebf81"
server_key_version: "encryptionkey@0"
I20260812 16:38:31.182646 3133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "d718d5598bf74e43ace4151e8792e42b"
format_stamp: "Formatted at 2026-08-12 16:38:31 on dist-test-slave-f7th"
server_key: "217c19279a3d623c16a77594d802d2eb"
server_key_iv: "831bdc3270f3cb7f31e4bdbdd58ebf81"
server_key_version: "encryptionkey@0"
I20260812 16:38:31.188282 3133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 16:38:31.192073 3148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:31.193558 3133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:31.193778 3133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "d718d5598bf74e43ace4151e8792e42b"
format_stamp: "Formatted at 2026-08-12 16:38:31 on dist-test-slave-f7th"
server_key: "217c19279a3d623c16a77594d802d2eb"
server_key_iv: "831bdc3270f3cb7f31e4bdbdd58ebf81"
server_key_version: "encryptionkey@0"
I20260812 16:38:31.194017 3133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:31.210445 3133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:31.215371 3133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:31.215751 3133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:31.236274 3133 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:41757
I20260812 16:38:31.236315 3199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:41757 every 8 connection(s)
I20260812 16:38:31.237808 3133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:31.239740 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 3133
I20260812 16:38:31.239976 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:31.240773 31990 external_mini_cluster.cc:1428] Setting key 0b56330db01748163c8d5fbef228f8c1
I20260812 16:38:31.242699 3200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 16:38:31 dist-test-slave-f7th krb5kdc[3120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:31.257977 3200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b: Bootstrap starting.
I20260812 16:38:31.261404 3200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:31.262712 3200 log.cc:824] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:31.266119 3200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b: No bootstrap required, opened a new log
I20260812 16:38:31.266140 3203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:31.243000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.31.61.190:41757)
0812 16:38:31.243821 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:31.243833 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:31.243877 (+ 44us) server_negotiation.cc:407] Connection header received
0812 16:38:31.245558 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:31.245583 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:31.246002 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:31.246362 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:31.248565 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:31.250177 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:31.251971 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:31.252708 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:31.252872 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:31.258142 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:31.258153 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:31.258170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:31.258218 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:31.261313 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:31.262263 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:31.262270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:31.262277 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:31.262475 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:31.262945 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:31.262950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:31.262953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:31.263753 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:31.264086 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:31.264387 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 16:38:31.265011 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":339,"threads_started":1}
I20260812 16:38:31.273694 3200 raft_consensus.cc:348] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718d5598bf74e43ace4151e8792e42b" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41757 } }
I20260812 16:38:31.274077 3200 raft_consensus.cc:374] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:31.274173 3200 raft_consensus.cc:729] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d718d5598bf74e43ace4151e8792e42b, State: Initialized, Role: FOLLOWER
I20260812 16:38:31.274725 3200 consensus_queue.cc:260] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718d5598bf74e43ace4151e8792e42b" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41757 } }
I20260812 16:38:31.274950 3200 raft_consensus.cc:388] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:31.275072 3200 raft_consensus.cc:482] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:31.275249 3200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:31.277593 3200 raft_consensus.cc:504] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718d5598bf74e43ace4151e8792e42b" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41757 } }
I20260812 16:38:31.278076 3200 leader_election.cc:302] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d718d5598bf74e43ace4151e8792e42b; no voters:
I20260812 16:38:31.278667 3200 leader_election.cc:288] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:31.278798 3205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:31.279436 3205 raft_consensus.cc:686] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [term 1 LEADER]: Becoming Leader. State: Replica: d718d5598bf74e43ace4151e8792e42b, State: Running, Role: LEADER
I20260812 16:38:31.280048 3205 consensus_queue.cc:237] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718d5598bf74e43ace4151e8792e42b" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41757 } }
I20260812 16:38:31.280741 3200 sys_catalog.cc:564] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:31.283373 3207 sys_catalog.cc:455] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d718d5598bf74e43ace4151e8792e42b. Latest consensus state: current_term: 1 leader_uuid: "d718d5598bf74e43ace4151e8792e42b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718d5598bf74e43ace4151e8792e42b" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41757 } } }
I20260812 16:38:31.284495 3207 sys_catalog.cc:458] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:31.283211 3206 sys_catalog.cc:455] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d718d5598bf74e43ace4151e8792e42b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d718d5598bf74e43ace4151e8792e42b" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41757 } } }
I20260812 16:38:31.284911 3206 sys_catalog.cc:458] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:31.288232 3214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:31.293166 3214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:31.302471 3214 catalog_manager.cc:1347] Generated new cluster ID: 7a6ad146c8ae4fc6a3bb6cf6a4c2cad8
I20260812 16:38:31.302597 3214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:31.326821 3214 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:31.328828 3214 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:31.341341 3214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d718d5598bf74e43ace4151e8792e42b: Generated new TSK 0
I20260812 16:38:31.342298 3214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:38:31.396170 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41757
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:31.730901 3226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:31.731209 3226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:31.731333 3226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:31.738752 3226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:31.738888 3226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:31.739042 3226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:31.749112 3226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:31.750442 3226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:31.751771 3226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:31.764743 3231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:31.764650 3232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:31.764902 3234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:31.765616 3226 server_base.cc:1034] running on GCE node
I20260812 16:38:31.766252 3226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:31.767261 3226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:31.768504 3226 hybrid_clock.cc:648] HybridClock initialized: now 1786552711768445 us; error 43 us; skew 500 ppm
I20260812 16:38:31.768836 3226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:31 dist-test-slave-f7th krb5kdc[3120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552711, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:31.772059 3226 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:38:31.773479 3226 webserver.cc:460] Webserver started at http://127.31.61.129:43831/ using document root <none> and password file <none>
I20260812 16:38:31.774178 3226 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:31.774291 3226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:31.774587 3226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:31.777544 3226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "b42b6a1efe244d5aacca23f83cfb9c01"
format_stamp: "Formatted at 2026-08-12 16:38:31 on dist-test-slave-f7th"
server_key: "5031410f556ec75f9cf7a409e9d1212a"
server_key_iv: "874ee9c754cffc317109513c2ae66c4b"
server_key_version: "encryptionkey@0"
I20260812 16:38:31.778304 3226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "b42b6a1efe244d5aacca23f83cfb9c01"
format_stamp: "Formatted at 2026-08-12 16:38:31 on dist-test-slave-f7th"
server_key: "5031410f556ec75f9cf7a409e9d1212a"
server_key_iv: "874ee9c754cffc317109513c2ae66c4b"
server_key_version: "encryptionkey@0"
I20260812 16:38:31.783535 3226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 16:38:31.787216 3241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:31.788792 3226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:31.788981 3226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "b42b6a1efe244d5aacca23f83cfb9c01"
format_stamp: "Formatted at 2026-08-12 16:38:31 on dist-test-slave-f7th"
server_key: "5031410f556ec75f9cf7a409e9d1212a"
server_key_iv: "874ee9c754cffc317109513c2ae66c4b"
server_key_version: "encryptionkey@0"
I20260812 16:38:31.789198 3226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:31.811883 3226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:31.816229 3226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:31.816519 3226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:31.817523 3226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:31.818903 3226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:31.818988 3226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:31.819099 3226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:31.819149 3226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:31.849722 3226 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:39305
I20260812 16:38:31.849771 3353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:39305 every 8 connection(s)
I20260812 16:38:31.850996 3226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:31.859428 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 3226
I20260812 16:38:31.859669 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:31.860267 31990 external_mini_cluster.cc:1428] Setting key 7a1b6b257f44ed75b6dd8e23c3fb0b00
Aug 12 16:38:31 dist-test-slave-f7th krb5kdc[3120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552711, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:31.871954 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:31.853465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:53773 (local address 127.31.61.190:41757)
0812 16:38:31.853897 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:31.853901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:31.854738 (+ 837us) server_negotiation.cc:407] Connection header received
0812 16:38:31.856575 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:31.856580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:31.856661 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:31.856752 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:31.858212 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:31.859287 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:31.860969 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:31.861500 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:31.861606 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:31.865532 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:31.865539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:31.865544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:31.865584 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:31.868527 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:31.869540 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:31.869545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:31.869548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:31.869679 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:31.870179 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:31.870183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:31.870185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:31.870465 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:31.870582 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:31.871445 (+ 863us) server_negotiation.cc:299] Negotiation successful
0812 16:38:31.871683 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":325,"thread_start_us":219,"threads_started":1}
I20260812 16:38:31.873310 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:31.853878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:41757 (local address 127.31.61.129:53773)
0812 16:38:31.854581 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:31.854624 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:31.856300 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:31.856994 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:31.857002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:31.857512 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:31.858044 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:31.858060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:31.859453 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:31.859457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:31.860818 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:31.860834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:31.861761 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:31.861771 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:31.862102 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:31.862903 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:31.862930 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:31.865244 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:31.868701 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:31.868712 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:31.868732 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:31.869396 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:31.869784 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:31.869789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:31.869793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:31.870055 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:31.870581 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:31.870589 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:31.871034 (+ 445us) client_negotiation.cc:769] Sending connection context
0812 16:38:31.871328 (+ 294us) client_negotiation.cc:240] Negotiation successful
0812 16:38:31.871929 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":265,"threads_started":1}
I20260812 16:38:31.874972 3354 heartbeater.cc:347] Connected to a master server at 127.31.61.190:41757
I20260812 16:38:31.875284 3354 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:31.876052 3354 heartbeater.cc:511] Master 127.31.61.190:41757 requested a full tablet report, sending...
I20260812 16:38:31.878345 3165 ts_manager.cc:194] Registered new tserver with Master: b42b6a1efe244d5aacca23f83cfb9c01 (127.31.61.129:39305)
I20260812 16:38:31.879885 3165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:53773
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:38:31.908145 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41757
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:32.247114 3360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:32.247431 3360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:32.247574 3360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:32.255450 3360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:32.255636 3360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:32.255798 3360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:32.266059 3360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:32.267587 3360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:32.269060 3360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:32.282486 3365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:32.282464 3366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:32.282842 3368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:32.283398 3360 server_base.cc:1034] running on GCE node
I20260812 16:38:32.284147 3360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:32.285197 3360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:32.286438 3360 hybrid_clock.cc:648] HybridClock initialized: now 1786552712286382 us; error 45 us; skew 500 ppm
I20260812 16:38:32.286804 3360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:32 dist-test-slave-f7th krb5kdc[3120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552712, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:32.290505 3360 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:38:32.292606 3360 webserver.cc:460] Webserver started at http://127.31.61.130:36703/ using document root <none> and password file <none>
I20260812 16:38:32.293385 3360 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:32.293509 3360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:32.293841 3360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:32.297117 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "4973843e5a404105b725d97b8e583bab"
format_stamp: "Formatted at 2026-08-12 16:38:32 on dist-test-slave-f7th"
server_key: "8e9c1a74bf55efddb2181bad40b671b2"
server_key_iv: "d79d7d0cd9e90b2e1257ad2dda758d2d"
server_key_version: "encryptionkey@0"
I20260812 16:38:32.298074 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "4973843e5a404105b725d97b8e583bab"
format_stamp: "Formatted at 2026-08-12 16:38:32 on dist-test-slave-f7th"
server_key: "8e9c1a74bf55efddb2181bad40b671b2"
server_key_iv: "d79d7d0cd9e90b2e1257ad2dda758d2d"
server_key_version: "encryptionkey@0"
I20260812 16:38:32.306030 3360 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260812 16:38:32.311676 3376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:32.313453 3360 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:38:32.313661 3360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "4973843e5a404105b725d97b8e583bab"
format_stamp: "Formatted at 2026-08-12 16:38:32 on dist-test-slave-f7th"
server_key: "8e9c1a74bf55efddb2181bad40b671b2"
server_key_iv: "d79d7d0cd9e90b2e1257ad2dda758d2d"
server_key_version: "encryptionkey@0"
I20260812 16:38:32.313956 3360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:32.330605 3360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:32.335508 3360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:32.335836 3360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:32.336921 3360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:32.338526 3360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:32.338615 3360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:32.338716 3360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:32.338793 3360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:32.375192 3360 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:45201
I20260812 16:38:32.375227 3488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:45201 every 8 connection(s)
I20260812 16:38:32.376636 3360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:32.379971 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 3360
I20260812 16:38:32.380168 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:32.380637 31990 external_mini_cluster.cc:1428] Setting key a4b6305e957fc5f7983231876a9c5b98
Aug 12 16:38:32 dist-test-slave-f7th krb5kdc[3120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552712, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:32.399932 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:32.379696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:38051 (local address 127.31.61.190:41757)
0812 16:38:32.380233 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:32.380240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:32.381370 (+ 1130us) server_negotiation.cc:407] Connection header received
0812 16:38:32.383555 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:32.383559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:32.383660 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:32.383750 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:32.385791 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.386626 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.388711 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.389216 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.389355 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:32.393213 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:32.393221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:32.393225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:32.393268 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:32.396587 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.397431 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.397435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.397438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.397583 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.398049 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.398053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.398055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.398389 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:32.398525 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:32.399291 (+ 766us) server_negotiation.cc:299] Negotiation successful
0812 16:38:32.399524 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":260,"threads_started":1}
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
I20260812 16:38:32.401011 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:32.380317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:41757 (local address 127.31.61.130:38051)
0812 16:38:32.381189 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:32.381236 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:32.383234 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:32.384048 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:32.384061 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:32.384804 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:32.385601 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:32.385627 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.386786 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.386791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:32.388546 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:32.388561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.389474 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.389480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:32.389692 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:32.390364 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:32.390388 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:32.392909 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:32.396762 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:32.396771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:32.396787 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:32.397295 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:32.397698 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:32.397703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:32.397705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:32.397904 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:32.398525 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:32.398534 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:32.398897 (+ 363us) client_negotiation.cc:769] Sending connection context
0812 16:38:32.399163 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 16:38:32.399808 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":318,"threads_started":1}
I20260812 16:38:32.402627 3489 heartbeater.cc:347] Connected to a master server at 127.31.61.190:41757
I20260812 16:38:32.403016 3489 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
I20260812 16:38:32.403960 3489 heartbeater.cc:511] Master 127.31.61.190:41757 requested a full tablet report, sending...
I20260812 16:38:32.405566 3165 ts_manager.cc:194] Registered new tserver with Master: 4973843e5a404105b725d97b8e583bab (127.31.61.130:45201)
I20260812 16:38:32.406530 3165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:38051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:38:32.427119 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41757
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:32.754992 3496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:32.755290 3496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:32.755385 3496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:32.763350 3496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:32.763485 3496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:32.763636 3496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:32.774628 3496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:32.776024 3496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:32.777372 3496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:32.791990 3501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:32.792409 3502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:32.792757 3504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:32.793534 3496 server_base.cc:1034] running on GCE node
I20260812 16:38:32.794421 3496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:32.795450 3496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:32.797055 3496 hybrid_clock.cc:648] HybridClock initialized: now 1786552712796974 us; error 59 us; skew 500 ppm
I20260812 16:38:32.797406 3496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:32 dist-test-slave-f7th krb5kdc[3120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552712, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:32.800805 3496 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:38:32.802848 3496 webserver.cc:460] Webserver started at http://127.31.61.131:34051/ using document root <none> and password file <none>
I20260812 16:38:32.803740 3496 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:32.803882 3496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:32.804260 3496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:32.807464 3496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "3370e7978df84fb3af6e4fb445dd61b5"
format_stamp: "Formatted at 2026-08-12 16:38:32 on dist-test-slave-f7th"
server_key: "1abaf61f15bfe2b139c65c4bc11833f1"
server_key_iv: "40b9d7355e62c862c648a1599e24d5c4"
server_key_version: "encryptionkey@0"
I20260812 16:38:32.808357 3496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "3370e7978df84fb3af6e4fb445dd61b5"
format_stamp: "Formatted at 2026-08-12 16:38:32 on dist-test-slave-f7th"
server_key: "1abaf61f15bfe2b139c65c4bc11833f1"
server_key_iv: "40b9d7355e62c862c648a1599e24d5c4"
server_key_version: "encryptionkey@0"
I20260812 16:38:32.813716 3496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:38:32.818188 3511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:32.820371 3496 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:38:32.820566 3496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "3370e7978df84fb3af6e4fb445dd61b5"
format_stamp: "Formatted at 2026-08-12 16:38:32 on dist-test-slave-f7th"
server_key: "1abaf61f15bfe2b139c65c4bc11833f1"
server_key_iv: "40b9d7355e62c862c648a1599e24d5c4"
server_key_version: "encryptionkey@0"
I20260812 16:38:32.820781 3496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:32.837354 3496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:32.841365 3496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:32.841637 3496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:32.842576 3496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:32.843938 3496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:32.844014 3496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:32.844136 3496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:32.844197 3496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:32.877148 3496 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:32795
I20260812 16:38:32.877184 3624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:32795 every 8 connection(s)
I20260812 16:38:32.878537 3496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:32.883029 3354 heartbeater.cc:503] Master 127.31.61.190:41757 was elected leader, sending a full tablet report...
I20260812 16:38:32.888445 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 3496
I20260812 16:38:32.888633 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:32.889034 31990 external_mini_cluster.cc:1428] Setting key 3090dc353f95c89b13ec7661eb3219db
Aug 12 16:38:32 dist-test-slave-f7th krb5kdc[3120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552712, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:32.898600 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:32.881213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:56431 (local address 127.31.61.190:41757)
0812 16:38:32.881424 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:32.881431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:32.882470 (+ 1039us) server_negotiation.cc:407] Connection header received
0812 16:38:32.884651 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:32.884656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:32.884726 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:32.884831 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:32.886487 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.887241 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.888899 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.889328 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.889436 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:32.893191 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:32.893196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:32.893199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:32.893231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:32.895769 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.896483 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.896487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.896489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.896611 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.897002 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.897006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.897008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.897295 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:32.897472 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:32.898179 (+ 707us) server_negotiation.cc:299] Negotiation successful
0812 16:38:32.898344 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:38:32.899570 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:32.881577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:41757 (local address 127.31.61.131:56431)
0812 16:38:32.882306 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:32.882343 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:32.884318 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:32.885139 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:32.885151 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:32.885694 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:32.886317 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:32.886339 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.887394 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.887398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:32.888761 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:32.888775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.889561 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.889569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:32.889859 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:32.890574 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:32.890605 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:32.892983 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:32.895890 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:32.895897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:32.895911 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:32.896342 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:32.896698 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:32.896701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:32.896704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:32.896889 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:32.897435 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:32.897443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:32.897786 (+ 343us) client_negotiation.cc:769] Sending connection context
0812 16:38:32.898057 (+ 271us) client_negotiation.cc:240] Negotiation successful
0812 16:38:32.898482 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":275,"threads_started":1}
I20260812 16:38:32.900985 3625 heartbeater.cc:347] Connected to a master server at 127.31.61.190:41757
I20260812 16:38:32.901259 3625 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:32.902060 3625 heartbeater.cc:511] Master 127.31.61.190:41757 requested a full tablet report, sending...
I20260812 16:38:32.903541 3165 ts_manager.cc:194] Registered new tserver with Master: 3370e7978df84fb3af6e4fb445dd61b5 (127.31.61.131:32795)
I20260812 16:38:32.904353 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:38:32.904454 3165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:56431
Aug 12 16:38:32 dist-test-slave-f7th krb5kdc[3120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:38:32 dist-test-slave-f7th krb5kdc[3120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:32.953145 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:32.937104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.31.61.129:39305)
0812 16:38:32.937704 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:32.937712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:32.937739 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:38:32.937799 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:32.937805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:32.938017 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:32.938142 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:32.939883 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.940949 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.942340 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.943238 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.943356 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:32.947315 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:32.947333 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:32.947349 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:32.947390 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:32.950279 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.951051 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.951060 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.951062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.951198 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.951637 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.951642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.951645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.952013 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:32.952248 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:32.952622 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 16:38:32.952829 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":281,"threads_started":1}
Aug 12 16:38:32 dist-test-slave-f7th krb5kdc[3120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:32.971886 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:32.957590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.31.61.190:41757)
0812 16:38:32.957797 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:32.957803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:32.958001 (+ 198us) server_negotiation.cc:407] Connection header received
0812 16:38:32.958138 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:32.958142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:32.958216 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:32.958288 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:32.959480 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.960263 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.961706 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.962129 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.962222 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:32.966121 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:32.966127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:32.966130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:32.966168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:32.968963 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.969774 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.969778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.969781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.969911 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.970272 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.970275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.970277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.970632 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:32.970841 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:32.971220 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 16:38:32.971462 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 16:38:32.972435 3165 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46280
I20260812 16:38:32.974066 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 3226
I20260812 16:38:32.989512 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:32.975465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.31.61.129:39305)
0812 16:38:32.975719 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:32.975726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:32.975922 (+ 196us) server_negotiation.cc:407] Connection header received
0812 16:38:32.976061 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:32.976065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:32.976168 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:32.976248 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:32.977522 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.978364 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.980129 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:32.980650 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:32.980780 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:32.983400 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:32.983406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:32.983411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:32.983448 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:32.986730 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.987403 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.987407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.987409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.987631 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:32.988021 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:32.988025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:32.988027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:32.988390 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:32.988621 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:32.988972 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 16:38:32.989171 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 16:38:32.990197 3349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:33.074420 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 3226
I20260812 16:38:33.088999 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 3360
Aug 12 16:38:33 dist-test-slave-f7th krb5kdc[3120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:38:33.104682 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:33.089822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.31.61.130:45201)
0812 16:38:33.090343 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:33.090350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:33.090374 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:38:33.090432 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:33.090438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:33.090652 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:33.090796 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:33.092007 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:33.093092 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:33.094334 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:33.094859 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:33.094969 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:33.098977 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:33.098989 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:33.099009 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:33.099062 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:33.101909 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:33.102579 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:33.102587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:33.102589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:33.102713 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:33.103084 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:33.103088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:33.103090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:33.103437 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:33.103899 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:33.104079 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 16:38:33.104339 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":266,"threads_started":1}
I20260812 16:38:33.105950 3484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:33.188035 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 3360
I20260812 16:38:33.202286 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 3496
Aug 12 16:38:33 dist-test-slave-f7th krb5kdc[3120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:38:33.219023 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:33.203371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.31.61.131:32795)
0812 16:38:33.204032 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:33.204039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:33.204067 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:38:33.204124 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:33.204130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:33.204365 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:33.204497 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:33.205697 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:33.206738 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:33.208119 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:33.208741 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:33.208841 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:33.213039 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:33.213053 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:33.213069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:33.213106 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:33.216259 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:33.216964 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:33.216971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:33.216973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:33.217111 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:33.217517 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:33.217521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:33.217524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:33.217893 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:33.218148 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:33.218428 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 16:38:33.218658 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":356,"threads_started":1}
I20260812 16:38:33.220350 3620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:33.303392 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 3496
I20260812 16:38:33.317261 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 3133
I20260812 16:38:33.318531 3195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:33.421108 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 3133
2026-08-12T16:38:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3854 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:33 dist-test-slave-f7th krb5kdc[3644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:33 dist-test-slave-f7th krb5kdc[3644](info): set up 2 sockets
Aug 12 16:38:33 dist-test-slave-f7th krb5kdc[3644](info): commencing operation
krb5kdc: starting...
W20260812 16:38:34.447657 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:34 dist-test-slave-f7th krb5kdc[3644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:38:34.656509 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36821
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:34.983819 3657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:34.984126 3657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:34.984232 3657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:34.991918 3657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:34.992056 3657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:34.992102 3657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:34.992152 3657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:34.992193 3657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:35.003302 3657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:35.004850 3657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:35.006309 3657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:35.019799 3663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:35.019867 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:35.019835 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:35.020798 3657 server_base.cc:1034] running on GCE node
I20260812 16:38:35.021775 3657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:35.023552 3657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:35.024835 3657 hybrid_clock.cc:648] HybridClock initialized: now 1786552715024757 us; error 76 us; skew 500 ppm
I20260812 16:38:35.025291 3657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:35 dist-test-slave-f7th krb5kdc[3644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552715, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:35.028782 3657 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:38:35.030009 3657 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:35.030110 3657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:35.030396 3657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:35.033943 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "2a78772835764952a40772f40c006b6a"
format_stamp: "Formatted at 2026-08-12 16:38:35 on dist-test-slave-f7th"
server_key: "0fc8d58ba2e1f3f30774dddb7e9345ca"
server_key_iv: "e166c959ef1603f3ec2bb1b3254c13b3"
server_key_version: "encryptionkey@0"
I20260812 16:38:35.034797 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "2a78772835764952a40772f40c006b6a"
format_stamp: "Formatted at 2026-08-12 16:38:35 on dist-test-slave-f7th"
server_key: "0fc8d58ba2e1f3f30774dddb7e9345ca"
server_key_iv: "e166c959ef1603f3ec2bb1b3254c13b3"
server_key_version: "encryptionkey@0"
I20260812 16:38:35.041167 3657 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 16:38:35.046078 3671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:35.048448 3657 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 16:38:35.048648 3657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "2a78772835764952a40772f40c006b6a"
format_stamp: "Formatted at 2026-08-12 16:38:35 on dist-test-slave-f7th"
server_key: "0fc8d58ba2e1f3f30774dddb7e9345ca"
server_key_iv: "e166c959ef1603f3ec2bb1b3254c13b3"
server_key_version: "encryptionkey@0"
I20260812 16:38:35.048885 3657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:35.070914 3657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:35.075416 3657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:35.075749 3657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:35.094602 3657 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36821
I20260812 16:38:35.094620 3722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36821 every 8 connection(s)
I20260812 16:38:35.095547 3657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:35.097627 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 3657
I20260812 16:38:35.097839 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:35.098460 31990 external_mini_cluster.cc:1428] Setting key 25e2ffa188cbd9d92d5ef7f154b96fe0
I20260812 16:38:35.099421 3723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:35.114943 3723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a: Bootstrap starting.
Aug 12 16:38:35 dist-test-slave-f7th krb5kdc[3644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:35.119267 3723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:35.120697 3723 log.cc:824] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:35.123649 3723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a: No bootstrap required, opened a new log
I20260812 16:38:35.125038 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:35.100470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.31.61.190:36821)
0812 16:38:35.101641 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:35.101656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:35.101706 (+ 50us) server_negotiation.cc:407] Connection header received
0812 16:38:35.104188 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:35.104220 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:35.104782 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:35.105254 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:35.106879 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:35.108779 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:35.110623 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:35.111781 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:35.111985 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:35.116803 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:35.116815 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:35.116838 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:35.116898 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:35.120711 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:35.121531 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:35.121540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:35.121549 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:35.121758 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:35.122227 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:35.122234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:35.122237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:35.122805 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:35.123083 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:35.123639 (+ 556us) server_negotiation.cc:299] Negotiation successful
0812 16:38:35.124059 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":527,"threads_started":1}
I20260812 16:38:35.132208 3723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a78772835764952a40772f40c006b6a" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36821 } }
I20260812 16:38:35.132627 3723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:35.132722 3723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a78772835764952a40772f40c006b6a, State: Initialized, Role: FOLLOWER
I20260812 16:38:35.133301 3723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a78772835764952a40772f40c006b6a" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36821 } }
I20260812 16:38:35.133603 3723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:35.133792 3723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:35.133965 3723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:35.136626 3723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a78772835764952a40772f40c006b6a" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36821 } }
I20260812 16:38:35.137157 3723 leader_election.cc:302] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a78772835764952a40772f40c006b6a; no voters:
I20260812 16:38:35.137740 3723 leader_election.cc:288] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:35.137876 3728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:35.138521 3728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [term 1 LEADER]: Becoming Leader. State: Replica: 2a78772835764952a40772f40c006b6a, State: Running, Role: LEADER
I20260812 16:38:35.139086 3728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a78772835764952a40772f40c006b6a" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36821 } }
I20260812 16:38:35.140070 3723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:35.142112 3730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a78772835764952a40772f40c006b6a. Latest consensus state: current_term: 1 leader_uuid: "2a78772835764952a40772f40c006b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a78772835764952a40772f40c006b6a" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36821 } } }
I20260812 16:38:35.142531 3730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:35.143175 3729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a78772835764952a40772f40c006b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a78772835764952a40772f40c006b6a" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36821 } } }
I20260812 16:38:35.143527 3729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:35.147365 3736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:35.153009 3736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:35.162555 3736 catalog_manager.cc:1347] Generated new cluster ID: c7f1a14e4ca5471297cf270fc515a94d
I20260812 16:38:35.162683 3736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:35.190328 3736 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:35.192308 3736 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:35.204267 3736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a: Generated new TSK 0
I20260812 16:38:35.205392 3736 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:38:35.264010 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36821
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:35.604291 3749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:35.604593 3749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:35.604736 3749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:35.612037 3749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:35.612155 3749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:35.612303 3749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:35.622488 3749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:35.623888 3749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:35.625223 3749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:35.637880 3754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:35.637926 3755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:35.637903 3757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:35.639231 3749 server_base.cc:1034] running on GCE node
I20260812 16:38:35.639928 3749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:35.640868 3749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:35.642091 3749 hybrid_clock.cc:648] HybridClock initialized: now 1786552715642040 us; error 41 us; skew 500 ppm
I20260812 16:38:35.642398 3749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:35 dist-test-slave-f7th krb5kdc[3644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552715, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:35.645565 3749 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:38:35.646718 3749 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:35.646807 3749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:35.647280 3749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:35.651192 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "c4e3ed91a6e34b549e410ddf79c5497c"
format_stamp: "Formatted at 2026-08-12 16:38:35 on dist-test-slave-f7th"
server_key: "a4f8a915860303ae926c10bd59e5867d"
server_key_iv: "abb34184373dc1d0ee5d97dbd65492b7"
server_key_version: "encryptionkey@0"
I20260812 16:38:35.652211 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "c4e3ed91a6e34b549e410ddf79c5497c"
format_stamp: "Formatted at 2026-08-12 16:38:35 on dist-test-slave-f7th"
server_key: "a4f8a915860303ae926c10bd59e5867d"
server_key_iv: "abb34184373dc1d0ee5d97dbd65492b7"
server_key_version: "encryptionkey@0"
I20260812 16:38:35.657837 3749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:38:35.661942 3763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:35.663265 3749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:38:35.663431 3749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "c4e3ed91a6e34b549e410ddf79c5497c"
format_stamp: "Formatted at 2026-08-12 16:38:35 on dist-test-slave-f7th"
server_key: "a4f8a915860303ae926c10bd59e5867d"
server_key_iv: "abb34184373dc1d0ee5d97dbd65492b7"
server_key_version: "encryptionkey@0"
I20260812 16:38:35.663718 3749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:35.684796 3749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:35.689016 3749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:35.689289 3749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:35.690296 3749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:35.691746 3749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:35.691839 3749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:35.691951 3749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:35.692026 3749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:35.723104 3749 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:45147
I20260812 16:38:35.723146 3876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:45147 every 8 connection(s)
I20260812 16:38:35.724041 3749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:35.726828 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 3749
I20260812 16:38:35.727067 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:35.727708 31990 external_mini_cluster.cc:1428] Setting key 8ed2833fac292984b8463a9773cfac57
Aug 12 16:38:35 dist-test-slave-f7th krb5kdc[3644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552715, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:35.745819 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:35.726582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:42773 (local address 127.31.61.190:36821)
0812 16:38:35.727095 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:35.727101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:35.728084 (+ 983us) server_negotiation.cc:407] Connection header received
0812 16:38:35.730277 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:35.730282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:35.730369 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:35.730472 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:35.732097 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:35.733056 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:35.734654 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:35.735215 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:35.735316 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:35.739568 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:35.739574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:35.739577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:35.739639 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:35.742222 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:35.743165 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:35.743170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:35.743172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:35.743297 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:35.743843 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:35.743848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:35.743851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:35.744185 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:35.744311 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:35.745339 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0812 16:38:35.745541 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
I20260812 16:38:35.747550 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:35.727107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36821 (local address 127.31.61.129:42773)
0812 16:38:35.727910 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:35.727967 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:35.729995 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:35.730716 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:35.730725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:35.731251 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:35.731924 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:35.731944 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:35.733217 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:35.733221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:35.734500 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:35.734514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:35.735462 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:35.735469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:35.735781 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:35.736529 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:35.736561 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:35.739297 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:35.742374 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:35.742385 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:35.742403 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:35.743017 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:35.743409 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:35.743413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:35.743417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:35.743690 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:35.744328 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:35.744337 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:35.744822 (+ 485us) client_negotiation.cc:769] Sending connection context
0812 16:38:35.745182 (+ 360us) client_negotiation.cc:240] Negotiation successful
0812 16:38:35.745779 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":324,"threads_started":1}
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
I20260812 16:38:35.749509 3878 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36821
I20260812 16:38:35.749888 3878 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:35.750566 3878 heartbeater.cc:511] Master 127.31.61.190:36821 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
I20260812 16:38:35.753048 3688 ts_manager.cc:194] Registered new tserver with Master: c4e3ed91a6e34b549e410ddf79c5497c (127.31.61.129:45147)
I20260812 16:38:35.754498 3688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:42773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:38:35.778529 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36821
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:36.106887 3884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:36.107163 3884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:36.107288 3884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:36.114720 3884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:36.114847 3884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:36.115008 3884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:36.125032 3884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:36.126415 3884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:36.127753 3884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:36.140796 3889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:36.140906 3890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:36.140894 3892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:36.145625 3884 server_base.cc:1034] running on GCE node
I20260812 16:38:36.146473 3884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:36.147617 3884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:36.148865 3884 hybrid_clock.cc:648] HybridClock initialized: now 1786552716148818 us; error 33 us; skew 500 ppm
I20260812 16:38:36.149222 3884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:36 dist-test-slave-f7th krb5kdc[3644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552716, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:36.152693 3884 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:38:36.154191 3884 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:36.154305 3884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:36.154651 3884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:36.157994 3884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "f0ddb0406c2b447389ce969475472371"
format_stamp: "Formatted at 2026-08-12 16:38:36 on dist-test-slave-f7th"
server_key: "3f63cb2a4967167282263b9610f18a92"
server_key_iv: "1788d7b3cef664b6d9a9cd2d0e030da0"
server_key_version: "encryptionkey@0"
I20260812 16:38:36.158914 3884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "f0ddb0406c2b447389ce969475472371"
format_stamp: "Formatted at 2026-08-12 16:38:36 on dist-test-slave-f7th"
server_key: "3f63cb2a4967167282263b9610f18a92"
server_key_iv: "1788d7b3cef664b6d9a9cd2d0e030da0"
server_key_version: "encryptionkey@0"
I20260812 16:38:36.164965 3884 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 16:38:36.168854 3898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:36.170351 3884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 16:38:36.170542 3884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "f0ddb0406c2b447389ce969475472371"
format_stamp: "Formatted at 2026-08-12 16:38:36 on dist-test-slave-f7th"
server_key: "3f63cb2a4967167282263b9610f18a92"
server_key_iv: "1788d7b3cef664b6d9a9cd2d0e030da0"
server_key_version: "encryptionkey@0"
I20260812 16:38:36.170768 3884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:36.204795 3884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:36.209121 3884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:36.209391 3884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:36.210386 3884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:36.212087 3884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:36.212184 3884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:36.212301 3884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:36.212369 3884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:36.243422 3884 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:44011
I20260812 16:38:36.243486 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:44011 every 8 connection(s)
I20260812 16:38:36.244272 3884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:36.248909 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 3884
I20260812 16:38:36.249130 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:36.249622 31990 external_mini_cluster.cc:1428] Setting key 1549e100634d3c58a80c11bc3adba0b8
Aug 12 16:38:36 dist-test-slave-f7th krb5kdc[3644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552716, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:36.266666 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.247619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:57261 (local address 127.31.61.190:36821)
0812 16:38:36.248105 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:36.248110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:36.249439 (+ 1329us) server_negotiation.cc:407] Connection header received
0812 16:38:36.251494 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:36.251499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:36.251586 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:36.251727 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:36.253369 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.254404 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.256183 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.256999 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.257132 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.260809 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:36.260814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:36.260817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:36.260852 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:36.263421 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:36.264230 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:36.264235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:36.264237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:36.264377 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:36.264815 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:36.264819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:36.264821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:36.265097 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:36.265246 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:36.265967 (+ 721us) server_negotiation.cc:299] Negotiation successful
0812 16:38:36.266194 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260812 16:38:36.267755 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.248078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36821 (local address 127.31.61.130:57261)
0812 16:38:36.249247 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:36.249303 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:36.251234 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:36.251962 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:36.251972 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:36.252585 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:36.253167 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.253185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.254670 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.254675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.256028 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.256043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.257246 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.257252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.257462 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.258195 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:36.258219 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:36.260566 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:36.263574 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:36.263584 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:36.263626 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:36.264094 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:36.264487 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:36.264492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:36.264496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:36.264698 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:36.265223 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:36.265232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:36.265595 (+ 363us) client_negotiation.cc:769] Sending connection context
0812 16:38:36.265833 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 16:38:36.266400 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":333,"threads_started":1}
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
I20260812 16:38:36.269335 4011 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36821
I20260812 16:38:36.269631 4011 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:36.270294 4011 heartbeater.cc:511] Master 127.31.61.190:36821 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
I20260812 16:38:36.271826 3688 ts_manager.cc:194] Registered new tserver with Master: f0ddb0406c2b447389ce969475472371 (127.31.61.130:44011)
I20260812 16:38:36.272624 3688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:57261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:38:36.299362 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36821
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:36.630296 4017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:36.630618 4017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:36.630719 4017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:36.638293 4017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:36.638428 4017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:36.638566 4017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:36.649073 4017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:36.650404 4017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:36.651723 4017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:36.664458 4023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:36.664734 4025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:36.664424 4022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:36.669727 4017 server_base.cc:1034] running on GCE node
I20260812 16:38:36.670395 4017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:36.671420 4017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:36.672650 4017 hybrid_clock.cc:648] HybridClock initialized: now 1786552716672605 us; error 44 us; skew 500 ppm
I20260812 16:38:36.673022 4017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:36 dist-test-slave-f7th krb5kdc[3644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552716, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:36.676229 4017 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:38:36.677421 4017 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:36.677534 4017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:36.677846 4017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:36.680902 4017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "aede6230f4244816b9fa7b502de8df64"
format_stamp: "Formatted at 2026-08-12 16:38:36 on dist-test-slave-f7th"
server_key: "da0f81131d42d0cdd5303b3617fc0edf"
server_key_iv: "35753505da83816f9541d4a30a01b406"
server_key_version: "encryptionkey@0"
I20260812 16:38:36.681740 4017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "aede6230f4244816b9fa7b502de8df64"
format_stamp: "Formatted at 2026-08-12 16:38:36 on dist-test-slave-f7th"
server_key: "da0f81131d42d0cdd5303b3617fc0edf"
server_key_iv: "35753505da83816f9541d4a30a01b406"
server_key_version: "encryptionkey@0"
I20260812 16:38:36.686829 4017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:38:36.690351 4031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:36.691671 4017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:36.691825 4017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "aede6230f4244816b9fa7b502de8df64"
format_stamp: "Formatted at 2026-08-12 16:38:36 on dist-test-slave-f7th"
server_key: "da0f81131d42d0cdd5303b3617fc0edf"
server_key_iv: "35753505da83816f9541d4a30a01b406"
server_key_version: "encryptionkey@0"
I20260812 16:38:36.692027 4017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:36.711153 4017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:36.715382 4017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:36.715723 4017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:36.716702 4017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:36.718040 4017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:36.718134 4017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:36.718250 4017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:36.718325 4017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:36.751189 4017 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:33871
I20260812 16:38:36.751226 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:33871 every 8 connection(s)
I20260812 16:38:36.752076 4017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:36.757519 3878 heartbeater.cc:503] Master 127.31.61.190:36821 was elected leader, sending a full tablet report...
I20260812 16:38:36.761606 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 4017
I20260812 16:38:36.761797 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:36.762204 31990 external_mini_cluster.cc:1428] Setting key f025ab393768fae7ff1a111c3dd624f5
Aug 12 16:38:36 dist-test-slave-f7th krb5kdc[3644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552716, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:36.772804 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.754898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:60845 (local address 127.31.61.190:36821)
0812 16:38:36.755107 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:36.755112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:36.756283 (+ 1171us) server_negotiation.cc:407] Connection header received
0812 16:38:36.758387 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:36.758394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:36.758470 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:36.758552 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:36.760493 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.761286 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.763168 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.763648 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.763782 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.767379 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:36.767384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:36.767387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:36.767421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:36.769840 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:36.770546 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:36.770552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:36.770555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:36.770670 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:36.771074 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:36.771078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:36.771080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:36.771350 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:36.771483 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:36.772262 (+ 779us) server_negotiation.cc:299] Negotiation successful
0812 16:38:36.772482 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 16:38:36.773761 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.755304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36821 (local address 127.31.61.131:60845)
0812 16:38:36.756104 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:36.756160 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:36.758052 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:36.758838 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:36.758851 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:36.759641 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:36.760273 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.760298 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.761458 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.761462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.763019 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.763034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.763859 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.763864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.764127 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.764830 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:36.764854 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:36.767167 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:36.769975 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:36.769986 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:36.770004 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:36.770434 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:36.770779 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:36.770785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:36.770788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:36.770963 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:36.771474 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:36.771482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:36.771903 (+ 421us) client_negotiation.cc:769] Sending connection context
0812 16:38:36.772149 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 16:38:36.772705 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":337,"threads_started":1}
I20260812 16:38:36.775367 4145 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36821
I20260812 16:38:36.775770 4145 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:36.776485 4145 heartbeater.cc:511] Master 127.31.61.190:36821 requested a full tablet report, sending...
I20260812 16:38:36.777987 3688 ts_manager.cc:194] Registered new tserver with Master: aede6230f4244816b9fa7b502de8df64 (127.31.61.131:33871)
I20260812 16:38:36.778800 3688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:60845
I20260812 16:38:36.790524 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:38:36.808552 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.795353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.31.61.190:36821)
0812 16:38:36.795567 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:36.795574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:36.795992 (+ 418us) server_negotiation.cc:407] Connection header received
0812 16:38:36.796156 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:36.796161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:36.796232 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:36.796332 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:36.797946 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.798757 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.800224 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.800707 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.800813 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.803221 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:36.803226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:36.803230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:36.803282 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:36.805803 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:36.806468 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:36.806471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:36.806474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:36.806599 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:36.807025 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:36.807029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:36.807031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:36.807428 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:36.807696 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:36.808045 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 16:38:36.808218 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:38:36.813051 3688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:36.820060 3688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:36.855085 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.833433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.31.61.130:44011)
0812 16:38:36.845919 (+ 12486us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:36.845928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:36.845955 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:38:36.846047 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:36.846054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:36.846428 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:36.846662 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:36.847940 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.849535 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.851750 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.853765 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.853938 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.854140 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:36.854537 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 16:38:36.854724 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12315,"thread_start_us":12659,"threads_started":1}
I20260812 16:38:36.855252 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.833131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:33871 (local address 127.0.0.1:40032)
0812 16:38:36.844324 (+ 11193us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:36.844346 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:36.844494 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:36.845719 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:36.845727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:36.845758 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:36.846275 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.846295 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.848374 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.848379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.851964 (+ 3585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.851986 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.854081 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.854087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.854235 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.854320 (+ 85us) client_negotiation.cc:769] Sending connection context
0812 16:38:36.854456 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 16:38:36.854703 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11061,"mutex_wait_us":389,"thread_start_us":11188,"threads_started":1}
I20260812 16:38:36.855276 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.832979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:44011 (local address 127.0.0.1:53662)
0812 16:38:36.843777 (+ 10798us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:36.843844 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:36.844120 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:36.846679 (+ 2559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:36.846685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:36.846718 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:36.847748 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.847761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.849699 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.849706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.851570 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.851590 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.853972 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.853978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.854198 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.854321 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 16:38:36.854455 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 16:38:36.854703 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10186,"mutex_wait_us":826,"thread_start_us":11716,"threads_started":1}
I20260812 16:38:36.855933 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.833773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:45147 (local address 127.0.0.1:35254)
0812 16:38:36.844952 (+ 11179us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:36.844967 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:36.845072 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:36.846320 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:36.846330 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:36.846355 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:36.846885 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.846899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.848732 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.848739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.852106 (+ 3367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.852116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.854944 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.854951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.855170 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.855212 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 16:38:36.855319 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 16:38:36.855473 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11072,"mutex_wait_us":657,"thread_start_us":9223,"threads_started":1}
I20260812 16:38:36.856334 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.833655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.31.61.131:33871)
0812 16:38:36.845001 (+ 11346us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:36.845012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:36.845061 (+ 49us) server_negotiation.cc:407] Connection header received
0812 16:38:36.845173 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:36.845183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:36.845458 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:36.846375 (+ 917us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:36.846676 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.848221 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.852250 (+ 4029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.853617 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.853733 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.853951 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:36.855750 (+ 1799us) server_negotiation.cc:299] Negotiation successful
0812 16:38:36.855971 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11165,"thread_start_us":11006,"threads_started":1}
I20260812 16:38:36.856735 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.834312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.31.61.129:45147)
0812 16:38:36.845741 (+ 11429us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:36.845749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:36.845776 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:38:36.845865 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:36.845871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:36.846084 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:36.846919 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:36.847121 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.848554 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.852649 (+ 4095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.854374 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.854524 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.854764 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:36.856203 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0812 16:38:36.856407 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11149,"thread_start_us":10989,"threads_started":1}
I20260812 16:38:36.860563 3946 tablet_service.cc:1467] Processing CreateTablet for tablet b1bd670a773d41d28359995ef466250a (DEFAULT_TABLE table=test-table [id=948395d768014b3ba96fb15f71f76aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:36.860566 3811 tablet_service.cc:1467] Processing CreateTablet for tablet b1bd670a773d41d28359995ef466250a (DEFAULT_TABLE table=test-table [id=948395d768014b3ba96fb15f71f76aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:36.860561 4080 tablet_service.cc:1467] Processing CreateTablet for tablet b1bd670a773d41d28359995ef466250a (DEFAULT_TABLE table=test-table [id=948395d768014b3ba96fb15f71f76aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:36.861948 3946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bd670a773d41d28359995ef466250a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:36.862090 4080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bd670a773d41d28359995ef466250a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:36.862311 3811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bd670a773d41d28359995ef466250a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:36.873256 4163 tablet_bootstrap.cc:492] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: Bootstrap starting.
I20260812 16:38:36.876955 4164 tablet_bootstrap.cc:492] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: Bootstrap starting.
I20260812 16:38:36.877314 4163 tablet_bootstrap.cc:654] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:36.878708 4163 log.cc:824] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:36.880736 4164 tablet_bootstrap.cc:654] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:36.882110 4164 log.cc:824] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:36.882246 4163 tablet_bootstrap.cc:492] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: No bootstrap required, opened a new log
I20260812 16:38:36.882537 4163 ts_tablet_manager.cc:1397] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 16:38:36.883751 4165 tablet_bootstrap.cc:492] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: Bootstrap starting.
I20260812 16:38:36.885206 4164 tablet_bootstrap.cc:492] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: No bootstrap required, opened a new log
I20260812 16:38:36.885511 4164 ts_tablet_manager.cc:1397] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 16:38:36.886842 4165 tablet_bootstrap.cc:654] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:36.887856 4165 log.cc:824] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:36.890270 4165 tablet_bootstrap.cc:492] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: No bootstrap required, opened a new log
I20260812 16:38:36.890525 4165 ts_tablet_manager.cc:1397] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260812 16:38:36.892876 4163 raft_consensus.cc:348] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } }
I20260812 16:38:36.893342 4163 raft_consensus.cc:374] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:36.893507 4163 raft_consensus.cc:729] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ddb0406c2b447389ce969475472371, State: Initialized, Role: FOLLOWER
I20260812 16:38:36.894173 4163 consensus_queue.cc:260] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } }
I20260812 16:38:36.896126 4164 raft_consensus.cc:348] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } }
I20260812 16:38:36.896699 4164 raft_consensus.cc:374] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:36.896881 4164 raft_consensus.cc:729] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aede6230f4244816b9fa7b502de8df64, State: Initialized, Role: FOLLOWER
I20260812 16:38:36.896934 4011 heartbeater.cc:503] Master 127.31.61.190:36821 was elected leader, sending a full tablet report...
I20260812 16:38:36.897543 4164 consensus_queue.cc:260] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } }
I20260812 16:38:36.897775 4163 ts_tablet_manager.cc:1428] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260812 16:38:36.898730 4145 heartbeater.cc:503] Master 127.31.61.190:36821 was elected leader, sending a full tablet report...
I20260812 16:38:36.899422 4164 ts_tablet_manager.cc:1428] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260812 16:38:36.900583 4165 raft_consensus.cc:348] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } }
I20260812 16:38:36.901010 4165 raft_consensus.cc:374] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:36.901134 4165 raft_consensus.cc:729] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e3ed91a6e34b549e410ddf79c5497c, State: Initialized, Role: FOLLOWER
I20260812 16:38:36.901728 4165 consensus_queue.cc:260] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } }
I20260812 16:38:36.904337 4165 ts_tablet_manager.cc:1428] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260812 16:38:36.914490 4169 raft_consensus.cc:482] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:36.914850 4169 raft_consensus.cc:504] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } }
I20260812 16:38:36.916747 4169 leader_election.cc:288] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aede6230f4244816b9fa7b502de8df64 (127.31.61.131:33871), c4e3ed91a6e34b549e410ddf79c5497c (127.31.61.129:45147)
I20260812 16:38:36.923930 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.917333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:50201 (local address 127.31.61.129:45147)
0812 16:38:36.917563 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:36.917569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:36.918279 (+ 710us) server_negotiation.cc:407] Connection header received
0812 16:38:36.918501 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:36.918506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:36.918569 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:36.918650 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:36.919308 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.920125 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.922205 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.922958 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.923126 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.923205 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:36.923544 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 16:38:36.923692 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:38:36.924444 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.917289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:35241 (local address 127.31.61.131:33871)
0812 16:38:36.917512 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:36.917519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:36.918402 (+ 883us) server_negotiation.cc:407] Connection header received
0812 16:38:36.918579 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:36.918583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:36.918645 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:36.918719 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:36.919358 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.920132 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.922026 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.922951 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.923134 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.923209 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:36.923584 (+ 375us) server_negotiation.cc:299] Negotiation successful
0812 16:38:36.924224 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:38:36.924599 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.917390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:33871 (local address 127.31.61.130:35241)
0812 16:38:36.918275 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:36.918291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:36.918448 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:36.918745 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:36.918749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:36.918778 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:36.919205 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.919215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.920262 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.920266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.921873 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.921886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.923201 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.923209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.923371 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.923413 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 16:38:36.923498 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 16:38:36.923685 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":8,"thread_start_us":446,"threads_started":1}
I20260812 16:38:36.924621 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.917283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:45147 (local address 127.31.61.130:50201)
0812 16:38:36.918161 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:36.918180 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:36.918345 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:36.918678 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:36.918682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:36.918708 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:36.919160 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.919174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.920264 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.920270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.922032 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:36.922045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.923240 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.923247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:36.923396 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.923424 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 16:38:36.923501 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 16:38:36.923664 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":11,"thread_start_us":440,"threads_started":1}
I20260812 16:38:36.925767 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bd670a773d41d28359995ef466250a" candidate_uuid: "f0ddb0406c2b447389ce969475472371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aede6230f4244816b9fa7b502de8df64" is_pre_election: true
I20260812 16:38:36.925823 3831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bd670a773d41d28359995ef466250a" candidate_uuid: "f0ddb0406c2b447389ce969475472371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" is_pre_election: true
I20260812 16:38:36.926266 4100 raft_consensus.cc:2454] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0ddb0406c2b447389ce969475472371 in term 0.
I20260812 16:38:36.926291 3831 raft_consensus.cc:2454] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0ddb0406c2b447389ce969475472371 in term 0.
I20260812 16:38:36.926949 3900 leader_election.cc:302] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: aede6230f4244816b9fa7b502de8df64, c4e3ed91a6e34b549e410ddf79c5497c, f0ddb0406c2b447389ce969475472371; no voters:
I20260812 16:38:36.927349 4169 raft_consensus.cc:2781] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:36.927500 4169 raft_consensus.cc:482] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:36.927623 4169 raft_consensus.cc:3037] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:36.929768 4169 raft_consensus.cc:504] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } }
I20260812 16:38:36.930362 4169 leader_election.cc:288] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [CANDIDATE]: Term 1 election: Requested vote from peers aede6230f4244816b9fa7b502de8df64 (127.31.61.131:33871), c4e3ed91a6e34b549e410ddf79c5497c (127.31.61.129:45147)
I20260812 16:38:36.930847 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bd670a773d41d28359995ef466250a" candidate_uuid: "f0ddb0406c2b447389ce969475472371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aede6230f4244816b9fa7b502de8df64"
I20260812 16:38:36.931046 4100 raft_consensus.cc:3037] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:36.931052 3831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bd670a773d41d28359995ef466250a" candidate_uuid: "f0ddb0406c2b447389ce969475472371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4e3ed91a6e34b549e410ddf79c5497c"
I20260812 16:38:36.931283 3831 raft_consensus.cc:3037] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:36.933522 4100 raft_consensus.cc:2454] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0ddb0406c2b447389ce969475472371 in term 1.
I20260812 16:38:36.933622 3831 raft_consensus.cc:2454] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0ddb0406c2b447389ce969475472371 in term 1.
I20260812 16:38:36.933976 3900 leader_election.cc:302] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aede6230f4244816b9fa7b502de8df64, f0ddb0406c2b447389ce969475472371; no voters:
I20260812 16:38:36.934310 4169 raft_consensus.cc:2781] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:36.934744 4169 raft_consensus.cc:686] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 1 LEADER]: Becoming Leader. State: Replica: f0ddb0406c2b447389ce969475472371, State: Running, Role: LEADER
I20260812 16:38:36.935278 4169 consensus_queue.cc:237] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } }
I20260812 16:38:36.939951 3687 catalog_manager.cc:5495] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0ddb0406c2b447389ce969475472371 (127.31.61.130). New cstate: current_term: 1 leader_uuid: "f0ddb0406c2b447389ce969475472371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ddb0406c2b447389ce969475472371" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 } health_report { overall_health: UNKNOWN } } }
I20260812 16:38:36.959141 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:36.953059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.31.61.130:44011)
0812 16:38:36.953290 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:36.953295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:36.953386 (+ 91us) server_negotiation.cc:407] Connection header received
0812 16:38:36.953498 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:36.953502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:36.953622 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:36.953774 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:36.954201 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.954907 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.956498 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:36.957022 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:36.957138 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:36.957817 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:36.958401 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:36.958569 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:36.958810 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 16:38:36.958892 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 16:38:36.967702 3831 raft_consensus.cc:1261] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [term 1 FOLLOWER]: Refusing update from remote peer f0ddb0406c2b447389ce969475472371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:36.967702 4100 raft_consensus.cc:1261] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [term 1 FOLLOWER]: Refusing update from remote peer f0ddb0406c2b447389ce969475472371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:36.968696 4175 consensus_queue.cc:1035] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aede6230f4244816b9fa7b502de8df64" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:36.969101 4169 consensus_queue.cc:1035] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4e3ed91a6e34b549e410ddf79c5497c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:38:36.976400 3879 tablet.cc:2369] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:38:36.996708 3687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:36.997130 3687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:37.001394 3687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a78772835764952a40772f40c006b6a: Sending DeleteTablet for 3 replicas of tablet b1bd670a773d41d28359995ef466250a
I20260812 16:38:37.002700 3811 tablet_service.cc:1514] Processing DeleteTablet for tablet b1bd670a773d41d28359995ef466250a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:36 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:35254
I20260812 16:38:37.002764 4080 tablet_service.cc:1514] Processing DeleteTablet for tablet b1bd670a773d41d28359995ef466250a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:36 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:40032
I20260812 16:38:37.002859 3946 tablet_service.cc:1514] Processing DeleteTablet for tablet b1bd670a773d41d28359995ef466250a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:36 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:53662
I20260812 16:38:37.004567 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 3749
I20260812 16:38:37.016233 4190 tablet_replica.cc:330] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: stopping tablet replica
I20260812 16:38:37.016451 4189 tablet_replica.cc:330] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: stopping tablet replica
I20260812 16:38:37.016816 4190 raft_consensus.cc:2229] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:37.017607 4190 raft_consensus.cc:2258] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:37.017766 4187 tablet_replica.cc:330] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: stopping tablet replica
I20260812 16:38:37.018214 4187 raft_consensus.cc:2229] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:37.018774 4187 raft_consensus.cc:2258] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:37.019089 4189 raft_consensus.cc:2229] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:37.019665 4189 raft_consensus.cc:2258] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:37.020547 4190 ts_tablet_manager.cc:1905] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:37.021953 4189 ts_tablet_manager.cc:1905] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 16:38:37 dist-test-slave-f7th krb5kdc[3644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:37.028568 4187 ts_tablet_manager.cc:1905] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:37.029479 4189 ts_tablet_manager.cc:1918] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:37.029680 4189 log.cc:1196] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/b1bd670a773d41d28359995ef466250a
I20260812 16:38:37.030324 4189 ts_tablet_manager.cc:1939] T b1bd670a773d41d28359995ef466250a P c4e3ed91a6e34b549e410ddf79c5497c: Deleting consensus metadata
I20260812 16:38:37.032596 3675 catalog_manager.cc:4841] TS c4e3ed91a6e34b549e410ddf79c5497c (127.31.61.129:45147): tablet b1bd670a773d41d28359995ef466250a (table test-table [id=948395d768014b3ba96fb15f71f76aec]) successfully deleted
I20260812 16:38:37.034898 4190 ts_tablet_manager.cc:1918] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:37.035194 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:37.005947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.31.61.129:45147)
0812 16:38:37.006149 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:37.006155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:37.015638 (+ 9483us) server_negotiation.cc:407] Connection header received
0812 16:38:37.016703 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:37.016710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:37.016804 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:37.016912 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:37.018608 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:37.019719 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:37.021664 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:37.022298 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:37.022394 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:37.027517 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:37.027532 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:37.027564 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:37.027651 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:37.031487 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:37.032454 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:37.032464 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:37.032468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:37.032642 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:37.033103 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:37.033109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:37.033113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:37.033570 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:37.033876 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:37.034480 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 16:38:37.034735 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:38:37.035180 4190 log.cc:1196] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/b1bd670a773d41d28359995ef466250a
I20260812 16:38:37.035996 3872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:37.037281 4190 ts_tablet_manager.cc:1939] T b1bd670a773d41d28359995ef466250a P f0ddb0406c2b447389ce969475472371: Deleting consensus metadata
I20260812 16:38:37.038355 4187 ts_tablet_manager.cc:1918] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:37.038563 4187 log.cc:1196] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/b1bd670a773d41d28359995ef466250a
I20260812 16:38:37.039093 4187 ts_tablet_manager.cc:1939] T b1bd670a773d41d28359995ef466250a P aede6230f4244816b9fa7b502de8df64: Deleting consensus metadata
I20260812 16:38:37.039649 3674 catalog_manager.cc:4841] TS f0ddb0406c2b447389ce969475472371 (127.31.61.130:44011): tablet b1bd670a773d41d28359995ef466250a (table test-table [id=948395d768014b3ba96fb15f71f76aec]) successfully deleted
I20260812 16:38:37.041004 3673 catalog_manager.cc:4841] TS aede6230f4244816b9fa7b502de8df64 (127.31.61.131:33871): tablet b1bd670a773d41d28359995ef466250a (table test-table [id=948395d768014b3ba96fb15f71f76aec]) successfully deleted
I20260812 16:38:37.165998 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 3749
I20260812 16:38:37.181881 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 3884
Aug 12 16:38:37 dist-test-slave-f7th krb5kdc[3644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:38:37.197992 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:37.182729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.31.61.130:44011)
0812 16:38:37.182917 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:37.182923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:37.182942 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:38:37.183148 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:37.183152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:37.183229 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:37.183326 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:37.184630 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:37.185475 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:37.186773 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:37.187204 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:37.187322 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:37.191783 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:37.191801 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:37.191817 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:37.191864 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:37.194750 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:37.195814 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:37.195838 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:37.195842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:37.196015 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:37.196450 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:37.196454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:37.196456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:37.196903 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:37.197151 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:37.197486 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 16:38:37.197680 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:38:37.198789 4006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:37.314541 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 3884
I20260812 16:38:37.330241 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 4017
Aug 12 16:38:37 dist-test-slave-f7th krb5kdc[3644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:38:37.345705 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:37.331085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.31.61.131:33871)
0812 16:38:37.331295 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:37.331300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:37.331320 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:38:37.331451 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:37.331455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:37.331520 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:37.331634 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:37.332950 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:37.333924 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:37.335311 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:37.335793 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:37.335924 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:37.340141 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:37.340164 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:37.340185 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:37.340224 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:37.343112 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:37.343968 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:37.343976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:37.343978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:37.344122 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:37.344433 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:37.344438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:37.344440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:37.344804 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:37.345037 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:37.345292 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 16:38:37.345461 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 16:38:37.346385 4140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:37.435107 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 4017
I20260812 16:38:37.449235 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 3657
I20260812 16:38:37.450361 3718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:37.559373 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 3657
2026-08-12T16:38:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4138 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T16:38:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:37Z Disabled control of system clock
I20260812 16:38:37.623800 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:32981
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 16:38:37.945096 4199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:37.945405 4199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:37.945492 4199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:37.953374 4199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:37.953511 4199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:37.953564 4199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:37.953619 4199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:37.953662 4199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:37.963156 4199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:32981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:37.964447 4199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:37.966190 4199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:37.978368 4204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:37.978464 4205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:37.978432 4207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:37.979138 4199 server_base.cc:1034] running on GCE node
I20260812 16:38:37.980090 4199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:37.981678 4199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:37.982919 4199 hybrid_clock.cc:648] HybridClock initialized: now 1786552717982873 us; error 32 us; skew 500 ppm
I20260812 16:38:37.983300 4199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:38:37.985828 4199 webserver.cc:460] Webserver started at http://127.31.61.190:35503/ using document root <none> and password file <none>
I20260812 16:38:37.986572 4199 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:37.986671 4199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:37.987008 4199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:37.990198 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "4a86c995e625467c9864a041a7981644"
format_stamp: "Formatted at 2026-08-12 16:38:37 on dist-test-slave-f7th"
server_key: "46d950a1d8a421b3a5693cf1941bba4a"
server_key_iv: "2feac755ea5488817de8da5d4b2967c6"
server_key_version: "encryptionkey@0"
I20260812 16:38:37.991108 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "4a86c995e625467c9864a041a7981644"
format_stamp: "Formatted at 2026-08-12 16:38:37 on dist-test-slave-f7th"
server_key: "46d950a1d8a421b3a5693cf1941bba4a"
server_key_iv: "2feac755ea5488817de8da5d4b2967c6"
server_key_version: "encryptionkey@0"
I20260812 16:38:37.996905 4199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 16:38:38.000686 4213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:38.002452 4199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:38.002652 4199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "4a86c995e625467c9864a041a7981644"
format_stamp: "Formatted at 2026-08-12 16:38:37 on dist-test-slave-f7th"
server_key: "46d950a1d8a421b3a5693cf1941bba4a"
server_key_iv: "2feac755ea5488817de8da5d4b2967c6"
server_key_version: "encryptionkey@0"
I20260812 16:38:38.002880 4199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:38.009039 4199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:38.009882 4199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:38.010165 4199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:38.027861 4199 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:32981
I20260812 16:38:38.027899 4264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:32981 every 8 connection(s)
I20260812 16:38:38.029229 4199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:38.033063 4265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:38.033728 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 4199
I20260812 16:38:38.033905 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:38.034525 31990 external_mini_cluster.cc:1428] Setting key 6cf37a8bf28e0b998f4316dbbe319060
I20260812 16:38:38.044173 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:38.036526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.31.61.190:32981)
0812 16:38:38.037755 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:38.037771 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:38.037825 (+ 54us) server_negotiation.cc:407] Connection header received
0812 16:38:38.039858 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:38.039889 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:38.040469 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:38.040983 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:38.042018 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:38.042027 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:38.042043 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:38.042062 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:38.042145 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:38.042170 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:38.042411 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:38.042567 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 16:38:38.042860 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":380,"threads_started":1}
I20260812 16:38:38.047855 4265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644: Bootstrap starting.
I20260812 16:38:38.051128 4265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:38.052605 4265 log.cc:824] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:38.055876 4265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644: No bootstrap required, opened a new log
I20260812 16:38:38.063711 4265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a86c995e625467c9864a041a7981644" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32981 } }
I20260812 16:38:38.064095 4265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:38.064193 4265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a86c995e625467c9864a041a7981644, State: Initialized, Role: FOLLOWER
I20260812 16:38:38.064756 4265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a86c995e625467c9864a041a7981644" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32981 } }
I20260812 16:38:38.064965 4265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:38.065096 4265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:38.065246 4265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:38.067588 4265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a86c995e625467c9864a041a7981644" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32981 } }
I20260812 16:38:38.068161 4265 leader_election.cc:302] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a86c995e625467c9864a041a7981644; no voters:
I20260812 16:38:38.068735 4265 leader_election.cc:288] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:38.068885 4270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:38.069657 4270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [term 1 LEADER]: Becoming Leader. State: Replica: 4a86c995e625467c9864a041a7981644, State: Running, Role: LEADER
I20260812 16:38:38.070212 4270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a86c995e625467c9864a041a7981644" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32981 } }
I20260812 16:38:38.071069 4265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:38.072741 4271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a86c995e625467c9864a041a7981644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a86c995e625467c9864a041a7981644" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32981 } } }
I20260812 16:38:38.073158 4271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:38.075346 4272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a86c995e625467c9864a041a7981644. Latest consensus state: current_term: 1 leader_uuid: "4a86c995e625467c9864a041a7981644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a86c995e625467c9864a041a7981644" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 32981 } } }
I20260812 16:38:38.075937 4272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:38.076692 4278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:38.082258 4278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:38.091219 4278 catalog_manager.cc:1347] Generated new cluster ID: bff9b1a881a44c9db5055384578d6ac1
I20260812 16:38:38.091347 4278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:38.110620 4278 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:38.111119 4278 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:38.129946 4278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644: Generated new TSK 0
I20260812 16:38:38.131217 4278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:38:38.148094 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32981
--builtin_ntp_servers=127.31.61.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 16:38:38.482709 4291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:38.483049 4291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:38.483184 4291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:38.491382 4291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:38.491524 4291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:38.491729 4291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:38.501884 4291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:38.503268 4291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:38.505134 4291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:38.519677 4296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:38.519739 4297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:38.519717 4300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:38.520552 4291 server_base.cc:1034] running on GCE node
I20260812 16:38:38.521179 4291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:38.522145 4291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:38.523368 4291 hybrid_clock.cc:648] HybridClock initialized: now 1786552718523304 us; error 47 us; skew 500 ppm
I20260812 16:38:38.523746 4291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:38:38.526057 4291 webserver.cc:460] Webserver started at http://127.31.61.129:39223/ using document root <none> and password file <none>
I20260812 16:38:38.527006 4291 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:38.527132 4291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:38.527501 4291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:38.530853 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "87a3b68a44384ca4876d41cff4900292"
format_stamp: "Formatted at 2026-08-12 16:38:38 on dist-test-slave-f7th"
server_key: "1d9a317e678a0bee94b120598786649b"
server_key_iv: "2b53a1b976fd808b7bc114a6a06d70e6"
server_key_version: "encryptionkey@0"
I20260812 16:38:38.532034 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "87a3b68a44384ca4876d41cff4900292"
format_stamp: "Formatted at 2026-08-12 16:38:38 on dist-test-slave-f7th"
server_key: "1d9a317e678a0bee94b120598786649b"
server_key_iv: "2b53a1b976fd808b7bc114a6a06d70e6"
server_key_version: "encryptionkey@0"
I20260812 16:38:38.538029 4291 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 16:38:38.542027 4306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:38.543731 4291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:38:38.543920 4291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "87a3b68a44384ca4876d41cff4900292"
format_stamp: "Formatted at 2026-08-12 16:38:38 on dist-test-slave-f7th"
server_key: "1d9a317e678a0bee94b120598786649b"
server_key_iv: "2b53a1b976fd808b7bc114a6a06d70e6"
server_key_version: "encryptionkey@0"
I20260812 16:38:38.544155 4291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:38.550043 4291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:38.551321 4291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:38.551781 4291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:38.552876 4291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:38.555006 4291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:38.555111 4291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:38.555217 4291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:38.555297 4291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:38.593490 4291 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:36997
I20260812 16:38:38.593541 4420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:36997 every 8 connection(s)
I20260812 16:38:38.595142 4291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:38.601579 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 4291
I20260812 16:38:38.601800 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:38.602602 31990 external_mini_cluster.cc:1428] Setting key 37b01b544da021c4be9b0a73adac4eb1
I20260812 16:38:38.606261 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32981
--builtin_ntp_servers=127.31.61.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 16:38:38.606662 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:38.599217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:45897 (local address 127.31.61.190:32981)
0812 16:38:38.599773 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:38.599780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:38.601816 (+ 2036us) server_negotiation.cc:407] Connection header received
0812 16:38:38.603948 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:38.603953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:38.604038 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:38.604135 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:38.605805 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:38.605810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:38.605813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:38.605830 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:38.605874 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:38.605886 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:38.605991 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:38.606280 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 16:38:38.606337 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":288,"threads_started":1}
I20260812 16:38:38.608150 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:38.599676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:32981 (local address 127.31.61.129:45897)
0812 16:38:38.601494 (+ 1818us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:38.601547 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:38.603668 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:38.604401 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:38.604410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:38.604521 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:38.605371 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:38:38.605376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:38.605442 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:38:38.605592 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:38.605994 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:38.606005 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:38.606014 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:38:38.606356 (+ 342us) client_negotiation.cc:240] Negotiation successful
0812 16:38:38.606740 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":428,"threads_started":1}
I20260812 16:38:38.610888 4421 heartbeater.cc:347] Connected to a master server at 127.31.61.190:32981
I20260812 16:38:38.611217 4421 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:38.611950 4421 heartbeater.cc:511] Master 127.31.61.190:32981 requested a full tablet report, sending...
I20260812 16:38:38.614280 4230 ts_manager.cc:194] Registered new tserver with Master: 87a3b68a44384ca4876d41cff4900292 (127.31.61.129:36997)
W20260812 16:38:38.964349 4425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:38.964686 4425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:38.964819 4425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:38.972805 4425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:38.972965 4425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:38.973130 4425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:38.983577 4425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:38.984953 4425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:38.986752 4425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:38.999974 4431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:39.000041 4433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:39.000195 4430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:39.001518 4425 server_base.cc:1034] running on GCE node
I20260812 16:38:39.002183 4425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:39.003170 4425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:39.004370 4425 hybrid_clock.cc:648] HybridClock initialized: now 1786552719004311 us; error 53 us; skew 500 ppm
I20260812 16:38:39.004688 4425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:38:39.007033 4425 webserver.cc:460] Webserver started at http://127.31.61.130:41213/ using document root <none> and password file <none>
I20260812 16:38:39.007786 4425 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:39.007891 4425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:39.008217 4425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:39.010954 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "a6bc5b451dd948589e0276d77a34fe92"
format_stamp: "Formatted at 2026-08-12 16:38:39 on dist-test-slave-f7th"
server_key: "e99ba8630279b473f89ff19178696bdc"
server_key_iv: "d7aaadcbed449e394dec26c72bfe7892"
server_key_version: "encryptionkey@0"
I20260812 16:38:39.011760 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "a6bc5b451dd948589e0276d77a34fe92"
format_stamp: "Formatted at 2026-08-12 16:38:39 on dist-test-slave-f7th"
server_key: "e99ba8630279b473f89ff19178696bdc"
server_key_iv: "d7aaadcbed449e394dec26c72bfe7892"
server_key_version: "encryptionkey@0"
I20260812 16:38:39.016763 4425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:38:39.020133 4439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:39.021662 4425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 16:38:39.021799 4425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "a6bc5b451dd948589e0276d77a34fe92"
format_stamp: "Formatted at 2026-08-12 16:38:39 on dist-test-slave-f7th"
server_key: "e99ba8630279b473f89ff19178696bdc"
server_key_iv: "d7aaadcbed449e394dec26c72bfe7892"
server_key_version: "encryptionkey@0"
I20260812 16:38:39.021976 4425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:39.027413 4425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:39.028265 4425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:39.028553 4425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:39.029508 4425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:39.030903 4425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:39.030997 4425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:39.031116 4425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:39.031183 4425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:39.066684 4555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:41285 every 8 connection(s)
I20260812 16:38:39.066763 4425 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:41285
I20260812 16:38:39.068645 4425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:39.070262 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 4425
I20260812 16:38:39.070412 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:39.070873 31990 external_mini_cluster.cc:1428] Setting key c3b1824928539e59d2b5dbbb524341f6
I20260812 16:38:39.074424 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32981
--builtin_ntp_servers=127.31.61.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 16:38:39.079079 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.073182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:46377 (local address 127.31.61.190:32981)
0812 16:38:39.074300 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:39.074306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:39.075193 (+ 887us) server_negotiation.cc:407] Connection header received
0812 16:38:39.076931 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:39.076937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:39.077004 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:39.077091 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:39.078385 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:39.078389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:39.078393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:39.078408 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:39.078440 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:39.078451 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:39.078560 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:39.078733 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 16:38:39.078801 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973}
I20260812 16:38:39.080124 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.073857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:32981 (local address 127.31.61.130:46377)
0812 16:38:39.075018 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:39.075081 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:39.076669 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:39.077339 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:39.077348 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:39.077443 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:39.078076 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:38:39.078078 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:39.078117 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:38:39.078207 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:39.078537 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:39.078547 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:39.078555 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:38:39.078809 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 16:38:39.079088 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":372,"threads_started":1}
I20260812 16:38:39.081471 4556 heartbeater.cc:347] Connected to a master server at 127.31.61.190:32981
I20260812 16:38:39.081780 4556 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:39.082391 4556 heartbeater.cc:511] Master 127.31.61.190:32981 requested a full tablet report, sending...
I20260812 16:38:39.083854 4230 ts_manager.cc:194] Registered new tserver with Master: a6bc5b451dd948589e0276d77a34fe92 (127.31.61.130:41285)
W20260812 16:38:39.427778 4559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:39.428092 4559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:39.428221 4559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:39.436510 4559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:39.436648 4559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:39.436807 4559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:39.446244 4559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:32981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:39.447765 4559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:39.450320 4559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:39.467236 4567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:39.467236 4565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:39.468220 4564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:39.473150 4559 server_base.cc:1034] running on GCE node
I20260812 16:38:39.473920 4559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:39.475665 4559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:39.476946 4559 hybrid_clock.cc:648] HybridClock initialized: now 1786552719476851 us; error 64 us; skew 500 ppm
I20260812 16:38:39.477312 4559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:38:39.479941 4559 webserver.cc:460] Webserver started at http://127.31.61.131:39345/ using document root <none> and password file <none>
I20260812 16:38:39.481047 4559 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:39.481247 4559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:39.481778 4559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:39.485378 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "d06439f31fe74550b2188dda5a6704ff"
format_stamp: "Formatted at 2026-08-12 16:38:39 on dist-test-slave-f7th"
server_key: "adaeb767b68e829455f88ac3432c9a19"
server_key_iv: "9b89a95bb55d6b185a306f1d488c599f"
server_key_version: "encryptionkey@0"
I20260812 16:38:39.486402 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "d06439f31fe74550b2188dda5a6704ff"
format_stamp: "Formatted at 2026-08-12 16:38:39 on dist-test-slave-f7th"
server_key: "adaeb767b68e829455f88ac3432c9a19"
server_key_iv: "9b89a95bb55d6b185a306f1d488c599f"
server_key_version: "encryptionkey@0"
I20260812 16:38:39.491997 4559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:38:39.496054 4573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:39.497872 4559 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:38:39.498052 4559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "d06439f31fe74550b2188dda5a6704ff"
format_stamp: "Formatted at 2026-08-12 16:38:39 on dist-test-slave-f7th"
server_key: "adaeb767b68e829455f88ac3432c9a19"
server_key_iv: "9b89a95bb55d6b185a306f1d488c599f"
server_key_version: "encryptionkey@0"
I20260812 16:38:39.498291 4559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:39.504720 4559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:39.505721 4559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:39.506150 4559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:39.507314 4559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:39.508939 4559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:39.509042 4559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:39.509151 4559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:39.509224 4559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:39.543815 4559 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:34767
I20260812 16:38:39.543886 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:34767 every 8 connection(s)
I20260812 16:38:39.545331 4559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:39.547559 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 4559
I20260812 16:38:39.547745 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:39.548287 31990 external_mini_cluster.cc:1428] Setting key 87849d4d9ca4a8be7fd2a0e96906b033
I20260812 16:38:39.555555 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.547898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:37299 (local address 127.31.61.190:32981)
0812 16:38:39.548117 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:39.548123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:39.550078 (+ 1955us) server_negotiation.cc:407] Connection header received
0812 16:38:39.552792 (+ 2714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:39.552797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:39.552875 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:39.552965 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:39.554769 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:39.554775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:39.554780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:39.554802 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:39.554843 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:39.554855 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:39.554961 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:39.555196 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 16:38:39.555269 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:38:39.556869 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.548473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:32981 (local address 127.31.61.131:37299)
0812 16:38:39.549920 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:39.549975 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:39.552464 (+ 2489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:39.553219 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:39.553227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:39.553331 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:39.554377 (+ 1046us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:38:39.554379 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:39.554429 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:38:39.554563 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:39.554952 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:39.554963 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:39.554972 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:38:39.555281 (+ 309us) client_negotiation.cc:240] Negotiation successful
0812 16:38:39.555579 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":1162,"threads_started":1}
I20260812 16:38:39.561281 4687 heartbeater.cc:347] Connected to a master server at 127.31.61.190:32981
I20260812 16:38:39.561702 4687 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:39.562485 4687 heartbeater.cc:511] Master 127.31.61.190:32981 requested a full tablet report, sending...
I20260812 16:38:39.564488 4230 ts_manager.cc:194] Registered new tserver with Master: d06439f31fe74550b2188dda5a6704ff (127.31.61.131:34767)
I20260812 16:38:39.565861 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:38:39.574249 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.571376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.31.61.190:32981)
0812 16:38:39.571554 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:39.571561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:39.572134 (+ 573us) server_negotiation.cc:407] Connection header received
0812 16:38:39.572272 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:39.572276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:39.572347 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:39.572458 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:39.573548 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:39.573554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:39.573559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:39.573578 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:39.573620 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:39.573635 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:39.573771 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:39.573815 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 16:38:39.573874 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 16:38:39.578823 4230 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:39.588860 4230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:39.617604 4421 heartbeater.cc:503] Master 127.31.61.190:32981 was elected leader, sending a full tablet report...
I20260812 16:38:39.624182 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.605497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:34767 (local address 127.0.0.1:50326)
0812 16:38:39.612566 (+ 7069us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:39.612625 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:39.612829 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:39.621145 (+ 8316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:39.621154 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:39.621278 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:39.622356 (+ 1078us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:38:39.622359 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:39.622377 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:38:39.622436 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:39.623393 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:39.623398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:39.623402 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 16:38:39.623549 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 16:38:39.623687 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6700,"thread_start_us":6490,"threads_started":1}
I20260812 16:38:39.624676 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.606388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.31.61.129:36997)
0812 16:38:39.619031 (+ 12643us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:39.619043 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:39.619089 (+ 46us) server_negotiation.cc:407] Connection header received
0812 16:38:39.622357 (+ 3268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:39.622367 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:39.622594 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:39.622730 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:39.623735 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:39.623742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:39.623750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:39.623764 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:39.623809 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:39.623826 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:39.623996 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:39.624202 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 16:38:39.624316 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12232,"thread_start_us":12717,"threads_started":1}
I20260812 16:38:39.624749 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.606207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:36997 (local address 127.0.0.1:53108)
0812 16:38:39.617124 (+ 10917us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:39.617151 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:39.622177 (+ 5026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:39.622741 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:39.622746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:39.622768 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:39.623493 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:38:39.623496 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:39.623513 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:38:39.623563 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:39.624048 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:39.624051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:39.624055 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 16:38:39.624213 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 16:38:39.624265 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10728,"thread_start_us":10536,"threads_started":1}
I20260812 16:38:39.624938 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.605778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.31.61.131:34767)
0812 16:38:39.619972 (+ 14194us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:39.619982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:39.620024 (+ 42us) server_negotiation.cc:407] Connection header received
0812 16:38:39.620111 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:39.620121 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:39.620417 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:39.620617 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:39.623033 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:39.623045 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:39.623057 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:39.623082 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:39.623143 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:39.623166 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:39.623334 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:39.624438 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0812 16:38:39.624553 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13958,"thread_start_us":14055,"threads_started":1}
I20260812 16:38:39.625248 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.613506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.31.61.130:41285)
0812 16:38:39.621014 (+ 7508us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:39.621021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:39.621043 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:38:39.621143 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:39.621149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:39.621301 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:39.621433 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:39.622535 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:39.622543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:39.622559 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:39.622591 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:39.622646 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:39.622666 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:39.622789 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:39.624828 (+ 2039us) server_negotiation.cc:299] Negotiation successful
0812 16:38:39.624941 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7359,"thread_start_us":7249,"threads_started":1}
I20260812 16:38:39.625772 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.613160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:41285 (local address 127.0.0.1:37172)
0812 16:38:39.618777 (+ 5617us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:39.618800 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:39.618924 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:39.621477 (+ 2553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:39.621482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:39.621518 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:39.622236 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:38:39.622240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:39.622283 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:38:39.622378 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:39.622850 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:39.622866 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:39.622874 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:38:39.624005 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0812 16:38:39.624062 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5468,"thread_start_us":5841,"threads_started":1}
I20260812 16:38:39.628748 4622 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30a889ff9746008ee15ff4465a9b23 (DEFAULT_TABLE table=test-table [id=54caa8cbcc604a29a66b7e035b0076c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:39.628961 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30a889ff9746008ee15ff4465a9b23 (DEFAULT_TABLE table=test-table [id=54caa8cbcc604a29a66b7e035b0076c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:39.629954 4622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30a889ff9746008ee15ff4465a9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:39.629446 4488 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30a889ff9746008ee15ff4465a9b23 (DEFAULT_TABLE table=test-table [id=54caa8cbcc604a29a66b7e035b0076c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:39.630430 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30a889ff9746008ee15ff4465a9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:39.630753 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30a889ff9746008ee15ff4465a9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:39.641361 4705 tablet_bootstrap.cc:492] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: Bootstrap starting.
I20260812 16:38:39.646109 4707 tablet_bootstrap.cc:492] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: Bootstrap starting.
I20260812 16:38:39.646299 4705 tablet_bootstrap.cc:654] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:39.647909 4705 log.cc:824] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:39.648485 4706 tablet_bootstrap.cc:492] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: Bootstrap starting.
I20260812 16:38:39.650357 4707 tablet_bootstrap.cc:654] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:39.651858 4707 log.cc:824] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:39.652434 4706 tablet_bootstrap.cc:654] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:39.653705 4706 log.cc:824] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:39.654363 4705 tablet_bootstrap.cc:492] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: No bootstrap required, opened a new log
I20260812 16:38:39.654680 4705 ts_tablet_manager.cc:1397] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260812 16:38:39.654898 4707 tablet_bootstrap.cc:492] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: No bootstrap required, opened a new log
I20260812 16:38:39.655148 4707 ts_tablet_manager.cc:1397] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260812 16:38:39.657002 4706 tablet_bootstrap.cc:492] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: No bootstrap required, opened a new log
I20260812 16:38:39.657279 4706 ts_tablet_manager.cc:1397] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260812 16:38:39.662432 4705 raft_consensus.cc:348] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } }
I20260812 16:38:39.662823 4705 raft_consensus.cc:374] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:39.662933 4705 raft_consensus.cc:729] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06439f31fe74550b2188dda5a6704ff, State: Initialized, Role: FOLLOWER
I20260812 16:38:39.663532 4705 consensus_queue.cc:260] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } }
I20260812 16:38:39.664690 4687 heartbeater.cc:503] Master 127.31.61.190:32981 was elected leader, sending a full tablet report...
I20260812 16:38:39.665400 4705 ts_tablet_manager.cc:1428] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260812 16:38:39.666769 4707 raft_consensus.cc:348] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } }
I20260812 16:38:39.667233 4707 raft_consensus.cc:374] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:39.667342 4707 raft_consensus.cc:729] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87a3b68a44384ca4876d41cff4900292, State: Initialized, Role: FOLLOWER
I20260812 16:38:39.667829 4706 raft_consensus.cc:348] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } }
I20260812 16:38:39.668319 4706 raft_consensus.cc:374] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:39.668157 4707 consensus_queue.cc:260] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } }
I20260812 16:38:39.668432 4706 raft_consensus.cc:729] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6bc5b451dd948589e0276d77a34fe92, State: Initialized, Role: FOLLOWER
I20260812 16:38:39.669335 4706 consensus_queue.cc:260] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } }
I20260812 16:38:39.672488 4707 ts_tablet_manager.cc:1428] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260812 16:38:39.676393 4556 heartbeater.cc:503] Master 127.31.61.190:32981 was elected leader, sending a full tablet report...
I20260812 16:38:39.676962 4706 ts_tablet_manager.cc:1428] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: Time spent starting tablet: real 0.019s user 0.014s sys 0.006s
W20260812 16:38:39.796959 4688 tablet.cc:2369] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:38:39.820981 4557 tablet.cc:2369] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:38:39.848717 4422 tablet.cc:2369] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:38:39.953573 4713 raft_consensus.cc:482] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:39.953889 4713 raft_consensus.cc:504] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } }
I20260812 16:38:39.956640 4713 leader_election.cc:288] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d06439f31fe74550b2188dda5a6704ff (127.31.61.131:34767), a6bc5b451dd948589e0276d77a34fe92 (127.31.61.130:41285)
I20260812 16:38:39.960472 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.957260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:35243 (local address 127.31.61.131:34767)
0812 16:38:39.957487 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:39.957495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:39.957971 (+ 476us) server_negotiation.cc:407] Connection header received
0812 16:38:39.958292 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:39.958298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:39.958361 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:39.958455 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:39.959861 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:39.959865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:39.959869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:39.959886 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:39.959919 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:39.959930 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:39.960055 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:39.960137 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 16:38:39.960202 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 16:38:39.960536 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.958174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:38673 (local address 127.31.61.130:41285)
0812 16:38:39.958354 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:39.958359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:39.958625 (+ 266us) server_negotiation.cc:407] Connection header received
0812 16:38:39.958719 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:39.958724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:39.958782 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:39.958893 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:39.959848 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:39.959853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:39.959856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:39.959871 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:39.959903 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:39.959915 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:39.960054 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:39.960136 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 16:38:39.960204 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 16:38:39.960842 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.957970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:41285 (local address 127.31.61.129:38673)
0812 16:38:39.958528 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:39.958542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:39.958634 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:39.958968 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:39.958971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:39.958989 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:39.959653 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:38:39.959657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:39.959672 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:38:39.959708 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:39.960025 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:39.960029 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:39.960034 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 16:38:39.960176 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 16:38:39.960232 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":271,"threads_started":1}
I20260812 16:38:39.960861 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.957198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:34767 (local address 127.31.61.129:35243)
0812 16:38:39.957866 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:39.957883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:39.958110 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:39.958663 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:39.958668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:39.958693 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:39.959501 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:38:39.959580 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:39.959649 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:38:39.959702 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:39.960031 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:39.960036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:39.960041 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 16:38:39.960243 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 16:38:39.960299 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":380,"threads_started":1}
I20260812 16:38:39.961908 4642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30a889ff9746008ee15ff4465a9b23" candidate_uuid: "87a3b68a44384ca4876d41cff4900292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06439f31fe74550b2188dda5a6704ff" is_pre_election: true
I20260812 16:38:39.961907 4509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30a889ff9746008ee15ff4465a9b23" candidate_uuid: "87a3b68a44384ca4876d41cff4900292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6bc5b451dd948589e0276d77a34fe92" is_pre_election: true
I20260812 16:38:39.962389 4642 raft_consensus.cc:2454] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87a3b68a44384ca4876d41cff4900292 in term 0.
I20260812 16:38:39.962381 4509 raft_consensus.cc:2454] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87a3b68a44384ca4876d41cff4900292 in term 0.
I20260812 16:38:39.963016 4310 leader_election.cc:302] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87a3b68a44384ca4876d41cff4900292, d06439f31fe74550b2188dda5a6704ff; no voters:
I20260812 16:38:39.963436 4713 raft_consensus.cc:2781] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:39.963621 4713 raft_consensus.cc:482] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:39.963733 4713 raft_consensus.cc:3037] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:39.966488 4713 raft_consensus.cc:504] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } }
I20260812 16:38:39.967337 4713 leader_election.cc:288] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [CANDIDATE]: Term 1 election: Requested vote from peers d06439f31fe74550b2188dda5a6704ff (127.31.61.131:34767), a6bc5b451dd948589e0276d77a34fe92 (127.31.61.130:41285)
I20260812 16:38:39.968021 4642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30a889ff9746008ee15ff4465a9b23" candidate_uuid: "87a3b68a44384ca4876d41cff4900292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06439f31fe74550b2188dda5a6704ff"
I20260812 16:38:39.968235 4642 raft_consensus.cc:3037] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:39.968312 4509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30a889ff9746008ee15ff4465a9b23" candidate_uuid: "87a3b68a44384ca4876d41cff4900292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6bc5b451dd948589e0276d77a34fe92"
I20260812 16:38:39.968667 4509 raft_consensus.cc:3037] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:39.970744 4642 raft_consensus.cc:2454] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87a3b68a44384ca4876d41cff4900292 in term 1.
I20260812 16:38:39.971154 4509 raft_consensus.cc:2454] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87a3b68a44384ca4876d41cff4900292 in term 1.
I20260812 16:38:39.971315 4310 leader_election.cc:302] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87a3b68a44384ca4876d41cff4900292, d06439f31fe74550b2188dda5a6704ff; no voters:
I20260812 16:38:39.971864 4713 raft_consensus.cc:2781] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:39.972579 4713 raft_consensus.cc:686] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 1 LEADER]: Becoming Leader. State: Replica: 87a3b68a44384ca4876d41cff4900292, State: Running, Role: LEADER
I20260812 16:38:39.973489 4713 consensus_queue.cc:237] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } }
I20260812 16:38:39.979176 4229 catalog_manager.cc:5495] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87a3b68a44384ca4876d41cff4900292 (127.31.61.129). New cstate: current_term: 1 leader_uuid: "87a3b68a44384ca4876d41cff4900292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87a3b68a44384ca4876d41cff4900292" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 36997 } health_report { overall_health: HEALTHY } } }
I20260812 16:38:40.001392 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:39.998547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.31.61.129:36997)
0812 16:38:39.998772 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:39.998779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:39.998918 (+ 139us) server_negotiation.cc:407] Connection header received
0812 16:38:39.999125 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:39.999129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:39.999219 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:39.999311 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:40.000698 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:40.000705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:40.000709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:40.000729 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:40.000766 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:40.000779 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:40.000919 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:40.001058 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 16:38:40.001138 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 16:38:40.010622 4509 raft_consensus.cc:1261] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [term 1 FOLLOWER]: Refusing update from remote peer 87a3b68a44384ca4876d41cff4900292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:40.010715 4642 raft_consensus.cc:1261] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [term 1 FOLLOWER]: Refusing update from remote peer 87a3b68a44384ca4876d41cff4900292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:40.011781 4717 consensus_queue.cc:1035] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6bc5b451dd948589e0276d77a34fe92" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:40.012202 4713 consensus_queue.cc:1035] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06439f31fe74550b2188dda5a6704ff" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:40.039983 4229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:40.040287 4229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:40.044659 4229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a86c995e625467c9864a041a7981644: Sending DeleteTablet for 3 replicas of tablet 6a30a889ff9746008ee15ff4465a9b23
I20260812 16:38:40.046099 4355 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a30a889ff9746008ee15ff4465a9b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:40 UTC) from {username='slave'} at 127.0.0.1:53108
I20260812 16:38:40.047150 4622 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a30a889ff9746008ee15ff4465a9b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:40 UTC) from {username='slave'} at 127.0.0.1:50326
I20260812 16:38:40.047832 4488 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a30a889ff9746008ee15ff4465a9b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:40 UTC) from {username='slave'} at 127.0.0.1:37172
I20260812 16:38:40.053721 4731 tablet_replica.cc:330] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: stopping tablet replica
I20260812 16:38:40.054265 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 4291
I20260812 16:38:40.054271 4731 raft_consensus.cc:2229] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:40.054878 4731 raft_consensus.cc:2258] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:40.055998 4732 tablet_replica.cc:330] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: stopping tablet replica
I20260812 16:38:40.056496 4732 raft_consensus.cc:2229] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:40.057060 4732 raft_consensus.cc:2258] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:40.057982 4730 tablet_replica.cc:330] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: stopping tablet replica
I20260812 16:38:40.058432 4730 raft_consensus.cc:2229] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:40.059162 4730 raft_consensus.cc:2258] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:40.061417 4731 ts_tablet_manager.cc:1905] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:40.064036 4732 ts_tablet_manager.cc:1905] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:40.064047 4730 ts_tablet_manager.cc:1905] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:40.068571 4731 ts_tablet_manager.cc:1918] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:40.068846 4731 log.cc:1196] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/6a30a889ff9746008ee15ff4465a9b23
I20260812 16:38:40.069411 4731 ts_tablet_manager.cc:1939] T 6a30a889ff9746008ee15ff4465a9b23 P d06439f31fe74550b2188dda5a6704ff: Deleting consensus metadata
I20260812 16:38:40.071578 4217 catalog_manager.cc:4841] TS d06439f31fe74550b2188dda5a6704ff (127.31.61.131:34767): tablet 6a30a889ff9746008ee15ff4465a9b23 (table test-table [id=54caa8cbcc604a29a66b7e035b0076c7]) successfully deleted
I20260812 16:38:40.072070 4732 ts_tablet_manager.cc:1918] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:40.072395 4732 log.cc:1196] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/6a30a889ff9746008ee15ff4465a9b23
I20260812 16:38:40.073042 4732 ts_tablet_manager.cc:1939] T 6a30a889ff9746008ee15ff4465a9b23 P a6bc5b451dd948589e0276d77a34fe92: Deleting consensus metadata
I20260812 16:38:40.074488 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.055667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.31.61.129:36997)
0812 16:38:40.055845 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:40.055851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:40.071575 (+ 15724us) server_negotiation.cc:407] Connection header received
0812 16:38:40.072126 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:40.072132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:40.072198 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:40.072273 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:40.073653 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:40.073660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:40.073664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:40.073684 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:40.073728 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:40.073745 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:40.073861 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:40.074115 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 16:38:40.074183 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:38:40.075026 4217 catalog_manager.cc:4841] TS a6bc5b451dd948589e0276d77a34fe92 (127.31.61.130:41285): tablet 6a30a889ff9746008ee15ff4465a9b23 (table test-table [id=54caa8cbcc604a29a66b7e035b0076c7]) successfully deleted
I20260812 16:38:40.075843 4730 ts_tablet_manager.cc:1918] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:40.076004 4730 log.cc:1196] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/6a30a889ff9746008ee15ff4465a9b23
I20260812 16:38:40.076052 4415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:40.177078 4730 ts_tablet_manager.cc:1939] T 6a30a889ff9746008ee15ff4465a9b23 P 87a3b68a44384ca4876d41cff4900292: Deleting consensus metadata
I20260812 16:38:40.177353 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 4291
W20260812 16:38:40.192615 4215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.61.129:36997 (error 108)
I20260812 16:38:40.193495 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 4425
I20260812 16:38:40.195142 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.194156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:36997 (local address 127.0.0.1:53130)
0812 16:38:40.194718 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:40.194929 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":311,"threads_started":1}
W20260812 16:38:40.195639 4215 catalog_manager.cc:4584] TS 87a3b68a44384ca4876d41cff4900292 (127.31.61.129:36997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a30a889ff9746008ee15ff4465a9b23: Network error: Client connection negotiation failed: client connection to 127.31.61.129:36997: connect: Connection refused (error 111)
I20260812 16:38:40.197474 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.194284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.31.61.130:41285)
0812 16:38:40.194485 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:40.194492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:40.194513 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:38:40.194703 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:40.194707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:40.194792 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:40.194875 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:40.196284 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:40.196289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:40.196293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:40.196310 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:40.196344 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:40.196357 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:40.196473 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:40.197076 (+ 603us) server_negotiation.cc:299] Negotiation successful
0812 16:38:40.197174 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 16:38:40.198544 4550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:40.250535 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.249425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:36997 (local address 127.0.0.1:53142)
0812 16:38:40.250057 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:40.250278 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366}
I20260812 16:38:40.252138 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.251473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:36997 (local address 127.0.0.1:53144)
0812 16:38:40.251741 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:40.251884 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 16:38:40.291898 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 4425
I20260812 16:38:40.309593 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 4559
I20260812 16:38:40.312683 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.312219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:36997 (local address 127.0.0.1:53152)
0812 16:38:40.312409 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:40.312492 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 16:38:40.312891 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.310397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.31.61.131:34767)
0812 16:38:40.310620 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:40.310628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:40.310650 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:38:40.310839 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:40.310845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:40.310960 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:40.311058 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:40.312118 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:40.312125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:40.312128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:40.312146 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:40.312181 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:40.312195 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:40.312325 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:40.312485 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 16:38:40.312553 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 16:38:40.313499 4682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:40.313748 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.313326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:36997 (local address 127.0.0.1:53154)
0812 16:38:40.313497 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:40.313586 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 16:38:40.394342 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.393842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:36997 (local address 127.0.0.1:53164)
0812 16:38:40.394112 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:40.394201 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 16:38:40.395493 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:40.395077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:36997 (local address 127.0.0.1:53178)
0812 16:38:40.395269 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:40.395342 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 16:38:40.401595 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 4559
I20260812 16:38:40.416260 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 4199
I20260812 16:38:40.417418 4260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:40.524901 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 4199
2026-08-12T16:38:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2965 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 16:38:40.572655 31990 mini_oidc.cc:152] Starting JWKS server
I20260812 16:38:40.575809 31990 webserver.cc:460] Webserver started at https://127.0.0.1:40181/ using document root <none> and password file <none>
I20260812 16:38:40.576184 31990 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 16:38:40.577051 31990 webserver.cc:460] Webserver started at http://127.0.0.1:45877/ using document root <none> and password file <none>
2026-08-12T16:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:40Z Disabled control of system clock
I20260812 16:38:40.606665 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:34769
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:34769
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40181/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/testchainca.cert with env {}
W20260812 16:38:40.962327 4745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:40.962638 4745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:40.962725 4745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:40.971649 4745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:40.971798 4745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:40.971853 4745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:40.971918 4745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:40.971967 4745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 16:38:40.972038 4745 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 16:38:40.972083 4745 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40181/jwks.json
I20260812 16:38:40.982929 4745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:34769
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:34769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40181/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:40.984285 4745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:40.985603 4745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:40.998476 4750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:40.998838 4753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:40.998699 4751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:41.000106 4745 server_base.cc:1034] running on GCE node
I20260812 16:38:41.001238 4745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:41.002926 4745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:41.004220 4745 hybrid_clock.cc:648] HybridClock initialized: now 1786552721004159 us; error 42 us; skew 500 ppm
I20260812 16:38:41.004714 4745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:38:41.007346 4745 webserver.cc:460] Webserver started at http://127.31.61.190:34655/ using document root <none> and password file <none>
I20260812 16:38:41.008244 4745 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:41.008359 4745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:41.008925 4745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:41.012346 4745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "f3e0cce97ab043db95cf36038584b913"
format_stamp: "Formatted at 2026-08-12 16:38:41 on dist-test-slave-f7th"
server_key: "2bc248ae1d3b22f4d9b1b26dca18180f"
server_key_iv: "377470bd4f87cd2eb5b981b3dc6d94a0"
server_key_version: "encryptionkey@0"
I20260812 16:38:41.013239 4745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "f3e0cce97ab043db95cf36038584b913"
format_stamp: "Formatted at 2026-08-12 16:38:41 on dist-test-slave-f7th"
server_key: "2bc248ae1d3b22f4d9b1b26dca18180f"
server_key_iv: "377470bd4f87cd2eb5b981b3dc6d94a0"
server_key_version: "encryptionkey@0"
I20260812 16:38:41.018774 4745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:38:41.023026 4759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:41.024802 4745 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:38:41.025027 4745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "f3e0cce97ab043db95cf36038584b913"
format_stamp: "Formatted at 2026-08-12 16:38:41 on dist-test-slave-f7th"
server_key: "2bc248ae1d3b22f4d9b1b26dca18180f"
server_key_iv: "377470bd4f87cd2eb5b981b3dc6d94a0"
server_key_version: "encryptionkey@0"
I20260812 16:38:41.025384 4745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:41.043864 4745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:41.044744 4745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:41.045037 4745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:41.063364 4745 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:34769
I20260812 16:38:41.063409 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:34769 every 8 connection(s)
I20260812 16:38:41.064899 4745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:41.068725 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:41.069144 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 4745
I20260812 16:38:41.069373 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:41.070174 31990 external_mini_cluster.cc:1428] Setting key 01e86284371108def39b9847e0323225
I20260812 16:38:41.082103 4813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913: Bootstrap starting.
I20260812 16:38:41.083993 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:41.072305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.31.61.190:34769)
0812 16:38:41.073289 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:41.073299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:41.073340 (+ 41us) server_negotiation.cc:407] Connection header received
0812 16:38:41.075261 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:41.075284 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:41.075825 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:41.076213 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:41.076815 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.078119 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.079443 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.080184 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.080369 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:41.081503 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:41.081511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:41.081525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:41.082065 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:41.082146 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:41.082166 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:41.082273 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:41.082603 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 16:38:41.082940 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":358,"threads_started":1}
I20260812 16:38:41.085927 4813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:41.087193 4813 log.cc:824] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:41.088189 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:38:41.090498 4813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913: No bootstrap required, opened a new log
I20260812 16:38:41.098665 4813 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e0cce97ab043db95cf36038584b913" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34769 } }
I20260812 16:38:41.099056 4813 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:41.099265 4813 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e0cce97ab043db95cf36038584b913, State: Initialized, Role: FOLLOWER
I20260812 16:38:41.099946 4813 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e0cce97ab043db95cf36038584b913" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34769 } }
I20260812 16:38:41.100170 4813 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:41.100263 4813 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:41.100392 4813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:41.102640 4813 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e0cce97ab043db95cf36038584b913" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34769 } }
I20260812 16:38:41.103170 4813 leader_election.cc:302] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3e0cce97ab043db95cf36038584b913; no voters:
I20260812 16:38:41.103806 4813 leader_election.cc:288] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:41.103978 4818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:41.105005 4818 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [term 1 LEADER]: Becoming Leader. State: Replica: f3e0cce97ab043db95cf36038584b913, State: Running, Role: LEADER
I20260812 16:38:41.105877 4818 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e0cce97ab043db95cf36038584b913" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34769 } }
I20260812 16:38:41.106277 4813 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:41.109167 4820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e0cce97ab043db95cf36038584b913. Latest consensus state: current_term: 1 leader_uuid: "f3e0cce97ab043db95cf36038584b913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e0cce97ab043db95cf36038584b913" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34769 } } }
I20260812 16:38:41.109249 4819 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3e0cce97ab043db95cf36038584b913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e0cce97ab043db95cf36038584b913" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34769 } } }
I20260812 16:38:41.109584 4820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:41.109622 4819 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:41.111225 4825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:41.117369 4825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:41.127590 4825 catalog_manager.cc:1347] Generated new cluster ID: 6c9e682f951644ee97074779aae74a07
I20260812 16:38:41.127756 4825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:41.150570 4825 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:41.152315 4825 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:41.170056 4825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3e0cce97ab043db95cf36038584b913: Generated new TSK 0
I20260812 16:38:41.171154 4825 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 16:38:41.231567 4843 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 16:38:41.231901 4843 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 16:38:41.234331 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:41.206345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.31.61.190:34769)
0812 16:38:41.206569 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:41.206577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:41.207224 (+ 647us) server_negotiation.cc:407] Connection header received
0812 16:38:41.207802 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:41.207807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:41.207926 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:41.208069 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0812 16:38:41.208698 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.209477 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.211705 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.212190 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.212305 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:41.213103 (+ 798us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 16:38:41.233714 (+ 20611us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 16:38:41.234076 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 16:38:41.234488 4816 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 16:38:41.236753 4842 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:38:41.206291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:34769 (local address 127.0.0.1:60058)
0812 16:38:41.207094 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:41.207115 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:41.207323 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:41.208136 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:41.208142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:41.208156 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0812 16:38:41.208526 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:41.208538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.209706 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.209712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:41.211488 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:41.211503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.212373 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.212380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:41.212568 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:41.212658 (+ 90us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 16:38:41.234107 (+ 21449us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 16:38:41.234495 (+ 388us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.61.190:34769: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":418,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.61.190:34769: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 16:38:41.238729 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 4745
I20260812 16:38:41.240216 4807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:41.340029 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 4745
2026-08-12T16:38:41Z chronyd exiting
I20260812 16:38:41.388799 31990 test_util.cc:175] -----------------------------------------------
I20260812 16:38:41.389034 31990 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786552676633950-31990-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (829 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 16:38:41.392851 31990 mini_oidc.cc:152] Starting JWKS server
I20260812 16:38:41.394716 31990 webserver.cc:460] Webserver started at https://127.0.0.1:42241/ using document root <none> and password file <none>
I20260812 16:38:41.394923 31990 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 16:38:41.395589 31990 webserver.cc:460] Webserver started at http://127.0.0.1:42183/ using document root <none> and password file <none>
2026-08-12T16:38:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:41Z Disabled control of system clock
I20260812 16:38:41.422528 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:38793
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:38793
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42241/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/testchainca.cert with env {}
W20260812 16:38:41.743216 4851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:41.743541 4851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:41.743898 4851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:41.751704 4851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:41.751847 4851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:41.751912 4851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:41.751992 4851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:41.752054 4851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 16:38:41.752128 4851 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 16:38:41.752182 4851 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42241/jwks.json
I20260812 16:38:41.763803 4851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:38793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:38793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42241/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:41.765206 4851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:41.766718 4851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:41.778571 4857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:41.778667 4856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:41.778652 4859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:41.779305 4851 server_base.cc:1034] running on GCE node
I20260812 16:38:41.780114 4851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:41.781569 4851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:41.782765 4851 hybrid_clock.cc:648] HybridClock initialized: now 1786552721782726 us; error 32 us; skew 500 ppm
I20260812 16:38:41.783094 4851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:38:41.789985 4851 webserver.cc:460] Webserver started at http://127.31.61.190:32797/ using document root <none> and password file <none>
I20260812 16:38:41.790681 4851 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:41.790792 4851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:41.791114 4851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:41.794023 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "cbdaa04c499a46fbad727936256cc666"
format_stamp: "Formatted at 2026-08-12 16:38:41 on dist-test-slave-f7th"
server_key: "2f3a00b1b22a08bc371d8b863ddca575"
server_key_iv: "f67333dcc883b74c24ec923ddf728240"
server_key_version: "encryptionkey@0"
I20260812 16:38:41.794790 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "cbdaa04c499a46fbad727936256cc666"
format_stamp: "Formatted at 2026-08-12 16:38:41 on dist-test-slave-f7th"
server_key: "2f3a00b1b22a08bc371d8b863ddca575"
server_key_iv: "f67333dcc883b74c24ec923ddf728240"
server_key_version: "encryptionkey@0"
I20260812 16:38:41.799975 4851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260812 16:38:41.803458 4865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:41.804813 4851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:41.804983 4851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "cbdaa04c499a46fbad727936256cc666"
format_stamp: "Formatted at 2026-08-12 16:38:41 on dist-test-slave-f7th"
server_key: "2f3a00b1b22a08bc371d8b863ddca575"
server_key_iv: "f67333dcc883b74c24ec923ddf728240"
server_key_version: "encryptionkey@0"
I20260812 16:38:41.805202 4851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:41.817147 4851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:41.818022 4851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:41.818332 4851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:41.837018 4851 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:38793
I20260812 16:38:41.837035 4918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:38793 every 8 connection(s)
I20260812 16:38:41.838421 4851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:41.842362 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 4851
I20260812 16:38:41.842437 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:41.842571 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:41.843376 31990 external_mini_cluster.cc:1428] Setting key 05102a9b980022961d37a1ac17f68f5f
I20260812 16:38:41.857533 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666: Bootstrap starting.
I20260812 16:38:41.858409 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:41.845371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.31.61.190:38793)
0812 16:38:41.846452 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:41.846464 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:41.846509 (+ 45us) server_negotiation.cc:407] Connection header received
0812 16:38:41.848606 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:41.848632 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:41.849107 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:41.849447 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:41.850200 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.851491 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.853447 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.854168 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.854308 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:41.856003 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:41.856027 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:41.856043 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:41.856608 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:41.856689 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:41.856711 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:41.856804 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:41.857061 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 16:38:41.857386 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":403,"threads_started":1}
I20260812 16:38:41.860992 4919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:41.862304 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:38:41.862268 4919 log.cc:824] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:41.865581 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666: No bootstrap required, opened a new log
I20260812 16:38:41.873256 4919 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa04c499a46fbad727936256cc666" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38793 } }
I20260812 16:38:41.873647 4919 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:41.873752 4919 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdaa04c499a46fbad727936256cc666, State: Initialized, Role: FOLLOWER
I20260812 16:38:41.874353 4919 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa04c499a46fbad727936256cc666" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38793 } }
I20260812 16:38:41.874575 4919 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:41.874706 4919 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:41.874848 4919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:41.877547 4919 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa04c499a46fbad727936256cc666" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38793 } }
I20260812 16:38:41.878057 4919 leader_election.cc:302] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbdaa04c499a46fbad727936256cc666; no voters:
I20260812 16:38:41.878705 4919 leader_election.cc:288] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:41.878795 4925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:41.879467 4925 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [term 1 LEADER]: Becoming Leader. State: Replica: cbdaa04c499a46fbad727936256cc666, State: Running, Role: LEADER
I20260812 16:38:41.880039 4925 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa04c499a46fbad727936256cc666" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38793 } }
I20260812 16:38:41.880892 4919 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:41.882661 4926 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbdaa04c499a46fbad727936256cc666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa04c499a46fbad727936256cc666" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38793 } } }
I20260812 16:38:41.882761 4927 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbdaa04c499a46fbad727936256cc666. Latest consensus state: current_term: 1 leader_uuid: "cbdaa04c499a46fbad727936256cc666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdaa04c499a46fbad727936256cc666" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38793 } } }
I20260812 16:38:41.883100 4926 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:41.883247 4927 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:41.885037 4930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:41.890287 4930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:41.900025 4930 catalog_manager.cc:1347] Generated new cluster ID: 4182d0f346104916acb66048b6ab2512
I20260812 16:38:41.900147 4930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:41.923877 4930 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:41.925681 4930 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:41.965162 4930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbdaa04c499a46fbad727936256cc666: Generated new TSK 0
I20260812 16:38:41.966179 4930 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 16:38:42.011271 4951 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 16:38:42.011684 4951 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 16:38:42.012737 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:41.991997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.31.61.190:38793)
0812 16:38:41.992194 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:41.992201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:41.992499 (+ 298us) server_negotiation.cc:407] Connection header received
0812 16:38:41.992733 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:41.992739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:41.992833 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:41.993016 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0812 16:38:41.993587 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.994719 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.996301 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.996975 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.997103 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:41.997650 (+ 547us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 16:38:42.012012 (+ 14362us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 16:38:42.012413 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 16:38:42.012820 4950 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:38:41.991831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:38793 (local address 127.0.0.1:55674)
0812 16:38:41.992394 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:41.992411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:41.992543 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:41.993091 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:41.993096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:41.993107 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0812 16:38:41.993402 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:41.993411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.994892 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.994897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:41.996142 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:41.996158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:41.997168 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:41.997173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:41.997309 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:41.997343 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 16:38:42.012273 (+ 14930us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 16:38:42.012511 (+ 238us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.61.190:38793: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":289,"threads_started":1}
W20260812 16:38:42.012867 4922 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260812 16:38:42.014221 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 4851
I20260812 16:38:42.015709 4914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:42.116901 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 4851
2026-08-12T16:38:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (780 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 16:38:42.173162 31990 mini_oidc.cc:152] Starting JWKS server
I20260812 16:38:42.175040 31990 webserver.cc:460] Webserver started at https://127.0.0.1:45589/ using document root <none> and password file <none>
I20260812 16:38:42.175282 31990 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 16:38:42.175948 31990 webserver.cc:460] Webserver started at http://127.0.0.1:36185/ using document root <none> and password file <none>
2026-08-12T16:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:42Z Disabled control of system clock
I20260812 16:38:42.204607 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:40329
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:40329
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45589/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 16:38:42.536320 4958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:42.536609 4958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:42.536693 4958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:42.543931 4958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:42.544034 4958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:42.544085 4958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:42.544140 4958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:42.544183 4958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 16:38:42.544241 4958 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 16:38:42.544279 4958 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45589/jwks.json
I20260812 16:38:42.553412 4958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:40329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:40329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45589/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:42.554729 4958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:42.556481 4958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:42.567988 4963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:42.568019 4964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:42.568045 4966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:42.568730 4958 server_base.cc:1034] running on GCE node
I20260812 16:38:42.569595 4958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:42.570971 4958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:42.572193 4958 hybrid_clock.cc:648] HybridClock initialized: now 1786552722572138 us; error 46 us; skew 500 ppm
I20260812 16:38:42.572640 4958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:38:42.575081 4958 webserver.cc:460] Webserver started at http://127.31.61.190:34735/ using document root <none> and password file <none>
I20260812 16:38:42.575834 4958 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:42.575937 4958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:42.576254 4958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:42.579183 4958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "5dbc780447d44a05afce545fa25182ff"
format_stamp: "Formatted at 2026-08-12 16:38:42 on dist-test-slave-f7th"
server_key: "b4df98a44e96a5de9a96f5ea64d59a1a"
server_key_iv: "a49839635e09ac0d20ea8a6b48529e1e"
server_key_version: "encryptionkey@0"
I20260812 16:38:42.579995 4958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "5dbc780447d44a05afce545fa25182ff"
format_stamp: "Formatted at 2026-08-12 16:38:42 on dist-test-slave-f7th"
server_key: "b4df98a44e96a5de9a96f5ea64d59a1a"
server_key_iv: "a49839635e09ac0d20ea8a6b48529e1e"
server_key_version: "encryptionkey@0"
I20260812 16:38:42.585050 4958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:38:42.588546 4972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:42.589968 4958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:42.590144 4958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "5dbc780447d44a05afce545fa25182ff"
format_stamp: "Formatted at 2026-08-12 16:38:42 on dist-test-slave-f7th"
server_key: "b4df98a44e96a5de9a96f5ea64d59a1a"
server_key_iv: "a49839635e09ac0d20ea8a6b48529e1e"
server_key_version: "encryptionkey@0"
I20260812 16:38:42.590349 4958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:42.603964 4958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:42.604851 4958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:42.605149 4958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:42.623466 4958 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:40329
I20260812 16:38:42.623488 5024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:40329 every 8 connection(s)
I20260812 16:38:42.624825 4958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:42.625530 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 4958
I20260812 16:38:42.625797 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:42.626618 31990 external_mini_cluster.cc:1428] Setting key 9ef5b28e64bc8ff4b0bcdfc04effb030
I20260812 16:38:42.629804 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:42.639910 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:42.628609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.31.61.190:40329)
0812 16:38:42.629679 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:42.629690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:42.629732 (+ 42us) server_negotiation.cc:407] Connection header received
0812 16:38:42.631363 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:42.631384 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:42.631857 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:42.632214 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:42.632797 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:42.634207 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:42.635555 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:42.636254 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:42.636421 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:42.637529 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:42.637536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:42.637557 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:42.638088 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:42.638168 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:42.638193 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:42.638295 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:42.638530 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 16:38:42.638859 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":401,"threads_started":1}
I20260812 16:38:42.641836 5025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff: Bootstrap starting.
I20260812 16:38:42.643874 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:38:42.645179 5025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:42.646289 5025 log.cc:824] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:42.649436 5025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff: No bootstrap required, opened a new log
I20260812 16:38:42.657680 5025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbc780447d44a05afce545fa25182ff" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40329 } }
I20260812 16:38:42.658098 5025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:42.658191 5025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dbc780447d44a05afce545fa25182ff, State: Initialized, Role: FOLLOWER
I20260812 16:38:42.658794 5025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbc780447d44a05afce545fa25182ff" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40329 } }
I20260812 16:38:42.659034 5025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:42.659163 5025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:42.659327 5025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:42.661993 5025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbc780447d44a05afce545fa25182ff" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40329 } }
I20260812 16:38:42.662814 5025 leader_election.cc:302] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dbc780447d44a05afce545fa25182ff; no voters:
I20260812 16:38:42.663546 5025 leader_election.cc:288] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:42.663751 5031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:42.664615 5031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [term 1 LEADER]: Becoming Leader. State: Replica: 5dbc780447d44a05afce545fa25182ff, State: Running, Role: LEADER
I20260812 16:38:42.665130 5031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbc780447d44a05afce545fa25182ff" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40329 } }
I20260812 16:38:42.666034 5025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:42.668589 5032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dbc780447d44a05afce545fa25182ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbc780447d44a05afce545fa25182ff" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40329 } } }
I20260812 16:38:42.669137 5032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:42.668602 5033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dbc780447d44a05afce545fa25182ff. Latest consensus state: current_term: 1 leader_uuid: "5dbc780447d44a05afce545fa25182ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbc780447d44a05afce545fa25182ff" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 40329 } } }
I20260812 16:38:42.669931 5033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:42.670591 5039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:42.676199 5039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:42.687414 5039 catalog_manager.cc:1347] Generated new cluster ID: 84421213606742928516f214fac10b5b
I20260812 16:38:42.687542 5039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:42.706038 5039 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:42.707501 5039 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:42.716682 5039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dbc780447d44a05afce545fa25182ff: Generated new TSK 0
I20260812 16:38:42.717576 5039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:38:42.749577 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:42.740264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.31.61.190:40329)
0812 16:38:42.740480 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:42.740487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:42.741089 (+ 602us) server_negotiation.cc:407] Connection header received
0812 16:38:42.741497 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:42.741619 (+ 122us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:42.741735 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:42.741899 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0812 16:38:42.742499 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:42.743280 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:42.745373 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:42.745851 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:42.745963 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:42.746696 (+ 733us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 16:38:42.748921 (+ 2225us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45589/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 16:38:42.749264 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45589/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 16:38:42.749706 5029 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45589/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 16:38:42.751907 5055 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:38:42.740241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:40329 (local address 127.0.0.1:53758)
0812 16:38:42.740940 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:42.740961 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:42.741150 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:42.741938 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:42.741944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:42.741958 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0812 16:38:42.742321 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:42.742333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:42.743422 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:42.743427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:42.745162 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:42.745179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:42.746058 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:42.746065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:42.746263 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:42.746312 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 16:38:42.749188 (+ 2876us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45589/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 16:38:42.751322 (+ 2134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.61.190:40329: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45589/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":1764,"thread_start_us":389,"threads_started":1}
I20260812 16:38:42.753015 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 4958
I20260812 16:38:42.754191 5020 generic_service.cc:196] Checking for leaks (request via RPC)
E20260812 16:38:42.761262 5056 webserver.cc:533] Webserver: sslize failed:
E20260812 16:38:42.761374 5056 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 16:38:42.846875 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 4958
2026-08-12T16:38:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (726 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T16:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:42Z Disabled control of system clock
I20260812 16:38:42.927184 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:33775
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:37093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:33775
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/testchainca.cert with env {}
W20260812 16:38:43.260987 5061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:43.261337 5061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:43.261438 5061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:43.268852 5061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:43.268976 5061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:43.269013 5061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:43.269062 5061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:43.269105 5061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 16:38:43.269167 5061 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 16:38:43.269209 5061 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 16:38:43.279732 5061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:37093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:33775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:43.281301 5061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:43.283440 5061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:43.297080 5066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:43.297034 5067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:43.297004 5069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:43.297808 5061 server_base.cc:1034] running on GCE node
I20260812 16:38:43.298646 5061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:43.300184 5061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:43.301482 5061 hybrid_clock.cc:648] HybridClock initialized: now 1786552723301426 us; error 50 us; skew 500 ppm
I20260812 16:38:43.301847 5061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:38:43.304329 5061 webserver.cc:460] Webserver started at http://127.31.61.190:46005/ using document root <none> and password file <none>
I20260812 16:38:43.305068 5061 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:43.305179 5061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:43.305482 5061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:43.308507 5061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f"
format_stamp: "Formatted at 2026-08-12 16:38:43 on dist-test-slave-f7th"
server_key: "4c5e007b43eb0e18e010fec505235c2b"
server_key_iv: "fa7c7e180397a330de58caa66f80b1d3"
server_key_version: "encryptionkey@0"
I20260812 16:38:43.309312 5061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f"
format_stamp: "Formatted at 2026-08-12 16:38:43 on dist-test-slave-f7th"
server_key: "4c5e007b43eb0e18e010fec505235c2b"
server_key_iv: "fa7c7e180397a330de58caa66f80b1d3"
server_key_version: "encryptionkey@0"
I20260812 16:38:43.315335 5061 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 16:38:43.319197 5075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:43.320662 5061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:43.320835 5061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f"
format_stamp: "Formatted at 2026-08-12 16:38:43 on dist-test-slave-f7th"
server_key: "4c5e007b43eb0e18e010fec505235c2b"
server_key_iv: "fa7c7e180397a330de58caa66f80b1d3"
server_key_version: "encryptionkey@0"
I20260812 16:38:43.321058 5061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:43.335412 5061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:43.336283 5061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:43.336560 5061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:43.354758 5061 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:33775
I20260812 16:38:43.354786 5127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:33775 every 8 connection(s)
I20260812 16:38:43.356277 5061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:43.358378 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 5061
I20260812 16:38:43.358560 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:43.359210 31990 external_mini_cluster.cc:1428] Setting key 66742a5169c12432ca3ad4ef2f097601
I20260812 16:38:43.360301 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:43.375806 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:43.361061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.31.61.190:33775)
0812 16:38:43.362338 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:43.362353 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:43.362409 (+ 56us) server_negotiation.cc:407] Connection header received
0812 16:38:43.364133 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:43.364164 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:43.364833 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:43.365312 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:43.366225 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:43.368363 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:43.370219 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:43.371358 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:43.371572 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:43.372780 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:43.372790 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:43.372811 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:38:43.373607 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:43.373705 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:38:43.373737 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:43.373853 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:43.374103 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 16:38:43.374449 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":554,"threads_started":1}
I20260812 16:38:43.375921 5128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f: Bootstrap starting.
I20260812 16:38:43.379448 5128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:43.379963 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:38:43.380676 5128 log.cc:824] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:43.383971 5128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f: No bootstrap required, opened a new log
I20260812 16:38:43.391466 5128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33775 } }
I20260812 16:38:43.391922 5128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:43.392030 5128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2344b43b0f4c0fbd757a5a2a43b77f, State: Initialized, Role: FOLLOWER
I20260812 16:38:43.392712 5128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33775 } }
I20260812 16:38:43.393122 5128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:43.393273 5128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:43.393452 5128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:43.395668 5128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33775 } }
I20260812 16:38:43.396196 5128 leader_election.cc:302] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a2344b43b0f4c0fbd757a5a2a43b77f; no voters:
I20260812 16:38:43.396775 5128 leader_election.cc:288] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:43.396983 5133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:43.397958 5133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [term 1 LEADER]: Becoming Leader. State: Replica: 2a2344b43b0f4c0fbd757a5a2a43b77f, State: Running, Role: LEADER
I20260812 16:38:43.398617 5133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33775 } }
I20260812 16:38:43.399070 5128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:43.401778 5134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33775 } } }
I20260812 16:38:43.401741 5135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a2344b43b0f4c0fbd757a5a2a43b77f. Latest consensus state: current_term: 1 leader_uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2344b43b0f4c0fbd757a5a2a43b77f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33775 } } }
I20260812 16:38:43.402179 5135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:43.402184 5134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:43.404286 5140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:43.409209 5140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:43.420159 5140 catalog_manager.cc:1347] Generated new cluster ID: 5be4d83495c54a7ab91d42a28d77ae8e
I20260812 16:38:43.420288 5140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:43.435392 5140 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:43.437271 5140 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:43.453800 5140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a2344b43b0f4c0fbd757a5a2a43b77f: Generated new TSK 0
I20260812 16:38:43.454991 5140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:38:43.556404 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:43.467711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.31.61.190:33775)
0812 16:38:43.467933 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:43.467940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:43.468435 (+ 495us) server_negotiation.cc:407] Connection header received
0812 16:38:43.468719 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:43.468724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:43.468803 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:43.468952 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0812 16:38:43.469623 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:43.470388 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:43.472605 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:43.473165 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:43.473290 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:43.473954 (+ 664us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 16:38:43.555632 (+ 81678us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 16:38:43.556079 (+ 447us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37520: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 16:38:43.556617 5158 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:38:43.467643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:33775 (local address 127.0.0.1:37520)
0812 16:38:43.468222 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:43.468246 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:43.468411 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:43.468980 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:43.468986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:43.468999 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0812 16:38:43.469307 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:43.469319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:43.470597 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:43.470603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:43.472329 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:43.472344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:43.473353 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:43.473360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:43.473559 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:43.473607 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 16:38:43.555989 (+ 82382us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 16:38:43.556245 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.61.190:33775: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":283,"threads_started":1}
W20260812 16:38:43.556715 5131 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37520: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260812 16:38:43.558418 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 5061
I20260812 16:38:43.559893 5123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:43.666004 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 5061
2026-08-12T16:38:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (800 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (417 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (439 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:44 dist-test-slave-f7th krb5kdc[5174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:44 dist-test-slave-f7th krb5kdc[5174](info): set up 2 sockets
Aug 12 16:38:44 dist-test-slave-f7th krb5kdc[5174](info): commencing operation
krb5kdc: starting...
W20260812 16:38:45.559924 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:45 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:38:45.780730 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:34501
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:46.110004 5187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:46.110304 5187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:46.110433 5187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:46.117841 5187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:46.117972 5187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:46.118028 5187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:46.118105 5187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:46.118155 5187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:46.129128 5187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:34501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:34501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:46.130643 5187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:46.132087 5187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:46.144629 5192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:46.144680 5195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:46.144743 5193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:46.146060 5187 server_base.cc:1034] running on GCE node
I20260812 16:38:46.147037 5187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:46.148731 5187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:46.149971 5187 hybrid_clock.cc:648] HybridClock initialized: now 1786552726149949 us; error 81 us; skew 500 ppm
I20260812 16:38:46.150444 5187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:46 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552726, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:46.153676 5187 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:38:46.155206 5187 webserver.cc:460] Webserver started at http://127.31.61.190:40305/ using document root <none> and password file <none>
I20260812 16:38:46.155973 5187 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:46.156100 5187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:46.156406 5187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:46.159459 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "f2927826bf2c4a34be4bed934094bfd6"
format_stamp: "Formatted at 2026-08-12 16:38:46 on dist-test-slave-f7th"
server_key: "2a77682429cbcc686f8b42c8e59a51c6"
server_key_iv: "9d6d6c10f9e6904f44c7c65909be12c4"
server_key_version: "encryptionkey@0"
I20260812 16:38:46.160336 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "f2927826bf2c4a34be4bed934094bfd6"
format_stamp: "Formatted at 2026-08-12 16:38:46 on dist-test-slave-f7th"
server_key: "2a77682429cbcc686f8b42c8e59a51c6"
server_key_iv: "9d6d6c10f9e6904f44c7c65909be12c4"
server_key_version: "encryptionkey@0"
I20260812 16:38:46.165829 5187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:38:46.169920 5202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:46.171458 5187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:38:46.171734 5187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "f2927826bf2c4a34be4bed934094bfd6"
format_stamp: "Formatted at 2026-08-12 16:38:46 on dist-test-slave-f7th"
server_key: "2a77682429cbcc686f8b42c8e59a51c6"
server_key_iv: "9d6d6c10f9e6904f44c7c65909be12c4"
server_key_version: "encryptionkey@0"
I20260812 16:38:46.171970 5187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:46.189855 5187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:46.194463 5187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:46.194798 5187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:46.215616 5187 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:34501
I20260812 16:38:46.215665 5256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:34501 every 8 connection(s)
I20260812 16:38:46.217094 5187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:46.220920 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:46.222779 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 5187
I20260812 16:38:46.223163 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:46.223877 31990 external_mini_cluster.cc:1428] Setting key 005d420e03e1e64245a168e2cfb07bec
I20260812 16:38:46.233515 5257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6: Bootstrap starting.
I20260812 16:38:46.237589 5257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:46.238956 5257 log.cc:824] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6: Log is configured to *not* fsync() on all Append() calls
Aug 12 16:38:46 dist-test-slave-f7th krb5kdc[5174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:46.242697 5257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6: No bootstrap required, opened a new log
I20260812 16:38:46.249406 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:46.226456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.31.61.190:34501)
0812 16:38:46.227328 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:46.227340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:46.227387 (+ 47us) server_negotiation.cc:407] Connection header received
0812 16:38:46.229136 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:46.229159 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:46.229579 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:46.229935 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:46.232004 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:46.233374 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:46.235169 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:46.236013 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:46.236164 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:46.241400 (+ 5236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:46.241409 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:46.241424 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:46.241464 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:46.244934 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:46.245868 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:46.245875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:46.245885 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:46.246095 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:46.246522 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:46.246526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:46.246528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:46.247125 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:46.247367 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:46.247897 (+ 530us) server_negotiation.cc:299] Negotiation successful
0812 16:38:46.248298 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":429,"threads_started":1}
I20260812 16:38:46.251432 5257 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2927826bf2c4a34be4bed934094bfd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34501 } }
I20260812 16:38:46.251848 5257 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:46.251946 5257 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2927826bf2c4a34be4bed934094bfd6, State: Initialized, Role: FOLLOWER
I20260812 16:38:46.252518 5257 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2927826bf2c4a34be4bed934094bfd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34501 } }
I20260812 16:38:46.252744 5257 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:46.252871 5257 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:46.253021 5257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:46.255247 5257 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2927826bf2c4a34be4bed934094bfd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34501 } }
I20260812 16:38:46.255788 5257 leader_election.cc:302] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2927826bf2c4a34be4bed934094bfd6; no voters:
I20260812 16:38:46.256358 5257 leader_election.cc:288] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:46.256513 5262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:46.257436 5262 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [term 1 LEADER]: Becoming Leader. State: Replica: f2927826bf2c4a34be4bed934094bfd6, State: Running, Role: LEADER
I20260812 16:38:46.258248 5262 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2927826bf2c4a34be4bed934094bfd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34501 } }
I20260812 16:38:46.258433 5257 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:46.261183 5263 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2927826bf2c4a34be4bed934094bfd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2927826bf2c4a34be4bed934094bfd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34501 } } }
I20260812 16:38:46.261157 5264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2927826bf2c4a34be4bed934094bfd6. Latest consensus state: current_term: 1 leader_uuid: "f2927826bf2c4a34be4bed934094bfd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2927826bf2c4a34be4bed934094bfd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34501 } } }
I20260812 16:38:46.261610 5263 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:46.261610 5264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:46.264720 5271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:46.269956 5271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:46.279282 5271 catalog_manager.cc:1347] Generated new cluster ID: d519a294ee83403ba83e5de0cf0634c0
I20260812 16:38:46.279392 5271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:46.298355 5271 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:46.299731 5271 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:46.314249 5271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2927826bf2c4a34be4bed934094bfd6: Generated new TSK 0
I20260812 16:38:46.315462 5271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:38:46.380228 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34501
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:46.706764 5284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:46.707069 5284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:46.707219 5284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:46.714505 5284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:46.714640 5284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:46.714795 5284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:46.724908 5284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:46.726256 5284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:46.727551 5284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:46.740919 5290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:46.741211 5289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:46.741194 5292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:46.741798 5284 server_base.cc:1034] running on GCE node
I20260812 16:38:46.742390 5284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:46.743366 5284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:46.744643 5284 hybrid_clock.cc:648] HybridClock initialized: now 1786552726744597 us; error 34 us; skew 500 ppm
I20260812 16:38:46.744982 5284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:46 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552726, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:46.748668 5284 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:38:46.750252 5284 webserver.cc:460] Webserver started at http://127.31.61.129:40279/ using document root <none> and password file <none>
I20260812 16:38:46.750990 5284 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:46.751147 5284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:46.751528 5284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:46.754776 5284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "f353e1d4c87341c9a03eebc930f8033e"
format_stamp: "Formatted at 2026-08-12 16:38:46 on dist-test-slave-f7th"
server_key: "d063a59880ec580333580802cbf7743f"
server_key_iv: "43d5ca9783f7ea570591ddb2ca0837b1"
server_key_version: "encryptionkey@0"
I20260812 16:38:46.755636 5284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "f353e1d4c87341c9a03eebc930f8033e"
format_stamp: "Formatted at 2026-08-12 16:38:46 on dist-test-slave-f7th"
server_key: "d063a59880ec580333580802cbf7743f"
server_key_iv: "43d5ca9783f7ea570591ddb2ca0837b1"
server_key_version: "encryptionkey@0"
I20260812 16:38:46.761368 5284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:38:46.765154 5299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:46.766547 5284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:46.766726 5284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "f353e1d4c87341c9a03eebc930f8033e"
format_stamp: "Formatted at 2026-08-12 16:38:46 on dist-test-slave-f7th"
server_key: "d063a59880ec580333580802cbf7743f"
server_key_iv: "43d5ca9783f7ea570591ddb2ca0837b1"
server_key_version: "encryptionkey@0"
I20260812 16:38:46.766929 5284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:46.787165 5284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:46.796353 5284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:46.796674 5284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:46.797734 5284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:46.799146 5284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:46.799227 5284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:46.799355 5284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:46.799407 5284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:46.832165 5284 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:46011
I20260812 16:38:46.832223 5412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:46011 every 8 connection(s)
I20260812 16:38:46.833504 5284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:46.843904 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 5284
I20260812 16:38:46.844120 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:46.844818 31990 external_mini_cluster.cc:1428] Setting key fa498fb2aac6722919722228e1dd5e15
Aug 12 16:38:46 dist-test-slave-f7th krb5kdc[5174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552726, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:46.854977 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:46.836107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:40179 (local address 127.31.61.190:34501)
0812 16:38:46.836593 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:46.836598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:46.837517 (+ 919us) server_negotiation.cc:407] Connection header received
0812 16:38:46.839432 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:46.839438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:46.839682 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:46.839802 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:46.841453 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:46.842179 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:46.844066 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:46.844558 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:46.844673 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:46.848907 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:46.848912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:46.848916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:46.848950 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:46.851426 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:46.852242 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:46.852246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:46.852248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:46.852370 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:46.852800 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:46.852803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:46.852806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:46.853084 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:46.853221 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:46.854346 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0812 16:38:46.854618 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":238,"threads_started":1}
I20260812 16:38:46.856395 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:46.836634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:34501 (local address 127.31.61.129:40179)
0812 16:38:46.837347 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:46.837385 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:46.839157 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:46.840101 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:46.840111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:46.840654 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:46.841264 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:46.841285 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:46.842347 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:46.842351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:46.843852 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:46.843871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:46.844822 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:46.844831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:46.845148 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:46.845871 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:46.845900 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:46.848671 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:46.851594 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:46.851632 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:46.851653 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:46.852120 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:46.852478 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:46.852483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:46.852486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:46.852682 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:46.853213 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:46.853221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:46.853583 (+ 362us) client_negotiation.cc:769] Sending connection context
0812 16:38:46.854215 (+ 632us) client_negotiation.cc:240] Negotiation successful
0812 16:38:46.854784 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":278,"threads_started":1}
I20260812 16:38:46.858279 5413 heartbeater.cc:347] Connected to a master server at 127.31.61.190:34501
I20260812 16:38:46.858649 5413 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:46.859688 5413 heartbeater.cc:511] Master 127.31.61.190:34501 requested a full tablet report, sending...
I20260812 16:38:46.862375 5221 ts_manager.cc:194] Registered new tserver with Master: f353e1d4c87341c9a03eebc930f8033e (127.31.61.129:46011)
I20260812 16:38:46.864078 5221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:40179
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:38:46.895767 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34501
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:47.230628 5419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:47.230903 5419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:47.230997 5419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:47.238435 5419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:47.238556 5419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:47.238690 5419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:47.249221 5419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:47.250614 5419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:47.252069 5419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:47.271023 5427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:47.271060 5425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:47.270994 5424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:47.272306 5419 server_base.cc:1034] running on GCE node
I20260812 16:38:47.273069 5419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:47.274152 5419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:47.275393 5419 hybrid_clock.cc:648] HybridClock initialized: now 1786552727275324 us; error 57 us; skew 500 ppm
I20260812 16:38:47.275835 5419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:47.279465 5419 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:38:47.281215 5419 webserver.cc:460] Webserver started at http://127.31.61.130:33821/ using document root <none> and password file <none>
I20260812 16:38:47.281973 5419 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:47.282105 5419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:47.282431 5419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:47.285625 5419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "79427fc273eb401f809a8bb378cf5697"
format_stamp: "Formatted at 2026-08-12 16:38:47 on dist-test-slave-f7th"
server_key: "ba42ad07808b4d52384cf21baedfd358"
server_key_iv: "031c79f677bfd93e4c62cbc9b4d09354"
server_key_version: "encryptionkey@0"
I20260812 16:38:47.286509 5419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "79427fc273eb401f809a8bb378cf5697"
format_stamp: "Formatted at 2026-08-12 16:38:47 on dist-test-slave-f7th"
server_key: "ba42ad07808b4d52384cf21baedfd358"
server_key_iv: "031c79f677bfd93e4c62cbc9b4d09354"
server_key_version: "encryptionkey@0"
I20260812 16:38:47.292119 5419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260812 16:38:47.295827 5434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:47.297335 5419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:47.297536 5419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "79427fc273eb401f809a8bb378cf5697"
format_stamp: "Formatted at 2026-08-12 16:38:47 on dist-test-slave-f7th"
server_key: "ba42ad07808b4d52384cf21baedfd358"
server_key_iv: "031c79f677bfd93e4c62cbc9b4d09354"
server_key_version: "encryptionkey@0"
I20260812 16:38:47.297782 5419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:47.326107 5419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:47.330590 5419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:47.330864 5419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:47.331948 5419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:47.333426 5419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:47.333575 5419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:47.333710 5419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:47.333767 5419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:47.368034 5419 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:40753
I20260812 16:38:47.368075 5548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:40753 every 8 connection(s)
I20260812 16:38:47.369318 5419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:47.377874 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 5419
I20260812 16:38:47.378064 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:47.378517 31990 external_mini_cluster.cc:1428] Setting key 9068872daaa167781266d83184f5f972
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:47.391112 5551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:47.371844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:33387 (local address 127.31.61.190:34501)
0812 16:38:47.372306 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:47.372313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:47.373394 (+ 1081us) server_negotiation.cc:407] Connection header received
0812 16:38:47.375232 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:47.375238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:47.375325 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:47.375420 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:47.377108 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.378018 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.379922 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.380480 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.380586 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:47.384382 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:47.384390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:47.384394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:47.384436 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:47.387703 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:47.388715 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:47.388719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:47.388722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:47.388855 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:47.389319 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:47.389322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:47.389324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:47.389604 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:47.389716 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:47.390628 (+ 912us) server_negotiation.cc:299] Negotiation successful
0812 16:38:47.390822 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260812 16:38:47.392494 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:47.372429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:34501 (local address 127.31.61.130:33387)
0812 16:38:47.373225 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:47.373269 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:47.374952 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:47.375749 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:47.375758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:47.376298 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:47.376929 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:47.376950 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.378204 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.378209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:47.379658 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:47.379674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.380739 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.380747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:47.380974 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:47.381651 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:47.381676 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:47.384094 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:47.387876 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:47.387887 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:47.387915 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:47.388578 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:47.388962 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:47.388967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:47.388970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:47.389214 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:47.389720 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:47.389729 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:47.390189 (+ 460us) client_negotiation.cc:769] Sending connection context
0812 16:38:47.390481 (+ 292us) client_negotiation.cc:240] Negotiation successful
0812 16:38:47.391073 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":349,"threads_started":1}
I20260812 16:38:47.394140 5549 heartbeater.cc:347] Connected to a master server at 127.31.61.190:34501
I20260812 16:38:47.394487 5549 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:47.395258 5549 heartbeater.cc:511] Master 127.31.61.190:34501 requested a full tablet report, sending...
I20260812 16:38:47.396979 5221 ts_manager.cc:194] Registered new tserver with Master: 79427fc273eb401f809a8bb378cf5697 (127.31.61.130:40753)
I20260812 16:38:47.397871 5221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:33387
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:38:47.429147 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34501
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:47.767055 5555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:47.767360 5555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:47.767485 5555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:47.775035 5555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:47.775169 5555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:47.775331 5555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:47.785974 5555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:47.787431 5555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:47.788869 5555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:47.802654 5560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:47.802627 5561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:47.802843 5563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:47.803469 5555 server_base.cc:1034] running on GCE node
I20260812 16:38:47.804175 5555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:47.805279 5555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:47.806524 5555 hybrid_clock.cc:648] HybridClock initialized: now 1786552727806437 us; error 64 us; skew 500 ppm
I20260812 16:38:47.806852 5555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:47.810314 5555 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:38:47.812278 5555 webserver.cc:460] Webserver started at http://127.31.61.131:36717/ using document root <none> and password file <none>
I20260812 16:38:47.812954 5555 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:47.813062 5555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:47.813342 5555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:47.816428 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "2e96b0ef31bf4358a879a8e56b7b16a4"
format_stamp: "Formatted at 2026-08-12 16:38:47 on dist-test-slave-f7th"
server_key: "8b3fee4e601c1189c44257d08c4fb2e6"
server_key_iv: "3c78b94caeca9668d07d3c84d8ad58a1"
server_key_version: "encryptionkey@0"
I20260812 16:38:47.817214 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "2e96b0ef31bf4358a879a8e56b7b16a4"
format_stamp: "Formatted at 2026-08-12 16:38:47 on dist-test-slave-f7th"
server_key: "8b3fee4e601c1189c44257d08c4fb2e6"
server_key_iv: "3c78b94caeca9668d07d3c84d8ad58a1"
server_key_version: "encryptionkey@0"
I20260812 16:38:47.822755 5555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:38:47.826467 5570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:47.828150 5555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:38:47.828315 5555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "2e96b0ef31bf4358a879a8e56b7b16a4"
format_stamp: "Formatted at 2026-08-12 16:38:47 on dist-test-slave-f7th"
server_key: "8b3fee4e601c1189c44257d08c4fb2e6"
server_key_iv: "3c78b94caeca9668d07d3c84d8ad58a1"
server_key_version: "encryptionkey@0"
I20260812 16:38:47.828524 5555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:47.855108 5555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:47.859385 5555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:47.859704 5555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:47.860755 5555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:47.862174 5555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:47.862257 5555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:47.862378 5555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:47.862423 5555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:47.867489 5413 heartbeater.cc:503] Master 127.31.61.190:34501 was elected leader, sending a full tablet report...
I20260812 16:38:47.895354 5555 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:39039
I20260812 16:38:47.895419 5684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:39039 every 8 connection(s)
I20260812 16:38:47.896649 5555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:47.900995 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 5555
I20260812 16:38:47.901165 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:47.901602 31990 external_mini_cluster.cc:1428] Setting key a115c4644a363ba3ee687dfaa66598cc
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:38:47.917703 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:47.899229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:46813 (local address 127.31.61.190:34501)
0812 16:38:47.899743 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:47.899749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:47.900700 (+ 951us) server_negotiation.cc:407] Connection header received
0812 16:38:47.902819 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:47.902824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:47.902912 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:47.903012 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:47.905133 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.905984 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.907539 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.908016 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.908113 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:47.911549 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:47.911555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:47.911558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:47.911592 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:47.914101 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:47.914864 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:47.914869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:47.914871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:47.915004 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:47.915667 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:47.915671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:47.915673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:47.916031 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:47.916181 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:47.917125 (+ 944us) server_negotiation.cc:299] Negotiation successful
0812 16:38:47.917327 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":218,"threads_started":1}
I20260812 16:38:47.918751 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:47.899801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:34501 (local address 127.31.61.131:46813)
0812 16:38:47.900540 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:47.900578 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:47.902567 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:47.903319 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:47.903332 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:47.904108 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:47.904922 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:47.904947 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.906152 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.906156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:47.907390 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:47.907402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.908219 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.908224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:47.908429 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:47.909056 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:47.909079 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:47.911279 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:47.914250 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:47.914258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:47.914273 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:47.914728 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:47.915221 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:47.915226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:47.915230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:47.915508 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:47.916205 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:47.916213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:47.916693 (+ 480us) client_negotiation.cc:769] Sending connection context
0812 16:38:47.916995 (+ 302us) client_negotiation.cc:240] Negotiation successful
0812 16:38:47.917536 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":289,"threads_started":1}
I20260812 16:38:47.920153 5685 heartbeater.cc:347] Connected to a master server at 127.31.61.190:34501
I20260812 16:38:47.920430 5685 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:47.921126 5685 heartbeater.cc:511] Master 127.31.61.190:34501 requested a full tablet report, sending...
I20260812 16:38:47.922557 5221 ts_manager.cc:194] Registered new tserver with Master: 2e96b0ef31bf4358a879a8e56b7b16a4 (127.31.61.131:39039)
I20260812 16:38:47.923424 5221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:46813
I20260812 16:38:47.930644 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/test-admin.keytab.
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:47.959308 31990 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 16:38:47.965021 5693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:38:47.962323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.31.61.129:46011)
0812 16:38:47.962953 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:47.962961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:47.962986 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:38:47.963045 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:47.963051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:47.963260 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:47.963436 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:47.964503 (+ 1067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33800: BlockingRecv error: recv got EOF from 127.0.0.1:33800 (error 108)
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":339,"threads_started":1}
W20260812 16:38:47.965024 31990 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5cc' of type FILE
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:47.966920 31990 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:47.968477 31990 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:47.984681 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:47.970268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.31.61.129:46011)
0812 16:38:47.970515 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:47.970521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:47.970662 (+ 141us) server_negotiation.cc:407] Connection header received
0812 16:38:47.970843 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:47.970847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:47.970941 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:47.971037 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:47.972153 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.973128 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.974351 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.974925 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.975041 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:47.978831 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:47.978846 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:47.978861 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:47.978898 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:47.981886 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:47.982588 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:47.982595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:47.982598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:47.982730 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:47.983081 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:47.983085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:47.983087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:47.983450 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:47.983769 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:47.984128 (+ 359us) server_negotiation.cc:299] Negotiation successful
0812 16:38:47.984390 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260812 16:38:47.990547 5693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:38:47.988793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.31.61.129:46011)
0812 16:38:47.989006 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:47.989012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:47.989222 (+ 210us) server_negotiation.cc:407] Connection header received
0812 16:38:47.989375 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:47.989380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:47.989440 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:47.989519 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:47.990294 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33822: BlockingRecv error: recv got EOF from 127.0.0.1:33822 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 16:38:47.990751 31990 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5cc' of type FILE
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:47.992442 31990 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:47.994005 31990 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 16:38:47 dist-test-slave-f7th krb5kdc[5174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:48.009943 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:47.995395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.31.61.129:46011)
0812 16:38:47.995693 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:47.995700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:47.995941 (+ 241us) server_negotiation.cc:407] Connection header received
0812 16:38:47.996035 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:47.996038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:47.996103 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:47.996182 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:47.997362 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.998087 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:47.999299 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:47.999870 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:48.000112 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:48.003914 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:48.003920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:48.003924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:48.003982 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:48.006683 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.007485 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.007490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.007493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.007703 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.008123 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.008129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.008133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.008679 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:48.009081 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:48.009372 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 16:38:48.009610 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260812 16:38:48.014371 5693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:38:48.012685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.31.61.129:46011)
0812 16:38:48.012879 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:48.012884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:48.013094 (+ 210us) server_negotiation.cc:407] Connection header received
0812 16:38:48.013214 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:48.013217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:48.013287 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:48.013365 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:48.014161 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33848: BlockingRecv error: recv got EOF from 127.0.0.1:33848 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 16:38:48.014683 31990 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786552676633950-31990-0/krb5kdc/krb5cc' of type FILE
Aug 12 16:38:48 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:48.016355 31990 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 16:38:48 dist-test-slave-f7th krb5kdc[5174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:48.017755 31990 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 16:38:48 dist-test-slave-f7th krb5kdc[5174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:38:48.031869 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:48.019160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.31.61.129:46011)
0812 16:38:48.019377 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:48.019384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:48.019729 (+ 345us) server_negotiation.cc:407] Connection header received
0812 16:38:48.019864 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:48.019867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:48.019928 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:48.020014 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:48.021058 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:48.021796 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:48.023088 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:48.023551 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:48.023705 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:48.027138 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:48.027143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:48.027146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:48.027180 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:48.029610 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.030230 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.030233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.030236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.030361 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.030663 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.030666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.030668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.030938 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:48.031161 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:48.031365 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 16:38:48.031540 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 16:38:48.033313 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 5284
I20260812 16:38:48.045472 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:48.033955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.31.61.129:46011)
0812 16:38:48.034132 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:48.034138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:48.034454 (+ 316us) server_negotiation.cc:407] Connection header received
0812 16:38:48.034568 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:48.034571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:48.034628 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:48.034699 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:48.035637 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:48.036681 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:48.037936 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:48.038323 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:48.038440 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:48.040428 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:48.040434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:48.040440 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:48.040472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:48.042713 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.043475 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.043480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.043482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.043650 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.044124 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.044128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.044130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.044423 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:48.044637 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:48.044949 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 16:38:48.045117 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:38:48.046175 5408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:48.130817 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 5284
I20260812 16:38:48.146163 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 5419
Aug 12 16:38:48 dist-test-slave-f7th krb5kdc[5174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:38:48.163879 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:48.147211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.31.61.130:40753)
0812 16:38:48.147858 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:48.147865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:48.147891 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:38:48.147951 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:48.147957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:48.148185 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:48.148312 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:48.149430 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:48.150448 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:48.152102 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:48.152654 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:48.152757 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:48.157040 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:48.157057 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:48.157072 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:48.157152 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:48.160283 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.161116 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.161124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.161127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.161282 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.161830 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.161835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.161837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.162373 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:48.162648 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:48.163118 (+ 470us) server_negotiation.cc:299] Negotiation successful
0812 16:38:48.163434 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":315,"threads_started":1}
I20260812 16:38:48.165228 5544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:48.251917 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 5419
I20260812 16:38:48.266191 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 5555
Aug 12 16:38:48 dist-test-slave-f7th krb5kdc[5174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:38:48.286135 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:48.267172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58724 (local address 127.31.61.131:39039)
0812 16:38:48.267928 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:48.267935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:48.267967 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:38:48.268030 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:48.268037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:48.268241 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:48.268373 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:48.269368 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:48.270587 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:48.271940 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:48.272670 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:48.272785 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:48.277155 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:48.277179 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:48.277203 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:48.277275 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:48.281212 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.282237 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.282248 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.282251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.282678 (+ 427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:48.283894 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:48.283901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:48.283904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:48.284708 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:48.285022 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:48.285399 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 16:38:48.285670 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":362,"threads_started":1}
I20260812 16:38:48.287577 5680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:48.374850 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 5555
I20260812 16:38:48.389541 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 5187
I20260812 16:38:48.390928 5252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:48.497676 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 5187
2026-08-12T16:38:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3979 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:48 dist-test-slave-f7th krb5kdc[5713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:48 dist-test-slave-f7th krb5kdc[5713](info): set up 2 sockets
Aug 12 16:38:48 dist-test-slave-f7th krb5kdc[5713](info): commencing operation
krb5kdc: starting...
W20260812 16:38:49.545369 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:49 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.190.keytab.
Entry for principal oryx/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.190.keytab.
I20260812 16:38:49.792204 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36585
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.190.keytab
--principal=oryx/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:50.142136 5727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:50.142583 5727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:50.142748 5727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:50.152266 5727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:50.152441 5727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:50.152506 5727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:50.152571 5727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:50.152621 5727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:50.163084 5727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.190.keytab
--principal=oryx/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:50.164539 5727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:50.165848 5727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:50.177796 5733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:50.177927 5732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:50.178092 5735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:50.179733 5727 server_base.cc:1034] running on GCE node
I20260812 16:38:50.180562 5727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:50.182168 5727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:50.183468 5727 hybrid_clock.cc:648] HybridClock initialized: now 1786552730183414 us; error 37 us; skew 500 ppm
I20260812 16:38:50.183872 5727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:50 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552730, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:50.187084 5727 init.cc:377] Logged in from keytab as oryx/127.31.61.190@KRBTEST.COM (short username oryx)
I20260812 16:38:50.188717 5727 webserver.cc:460] Webserver started at http://127.31.61.190:40593/ using document root <none> and password file <none>
I20260812 16:38:50.189409 5727 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:50.189532 5727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:50.189826 5727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:50.193075 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "f29b5c9b118c4339b003576414fa9bc9"
format_stamp: "Formatted at 2026-08-12 16:38:50 on dist-test-slave-f7th"
server_key: "f71aee67d79fdf57020f9ed3d2a508e0"
server_key_iv: "bcbf0f7d39c362a710ac21366723d29b"
server_key_version: "encryptionkey@0"
I20260812 16:38:50.193864 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "f29b5c9b118c4339b003576414fa9bc9"
format_stamp: "Formatted at 2026-08-12 16:38:50 on dist-test-slave-f7th"
server_key: "f71aee67d79fdf57020f9ed3d2a508e0"
server_key_iv: "bcbf0f7d39c362a710ac21366723d29b"
server_key_version: "encryptionkey@0"
I20260812 16:38:50.199111 5727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:38:50.203238 5742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:50.204704 5727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:50.204880 5727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "f29b5c9b118c4339b003576414fa9bc9"
format_stamp: "Formatted at 2026-08-12 16:38:50 on dist-test-slave-f7th"
server_key: "f71aee67d79fdf57020f9ed3d2a508e0"
server_key_iv: "bcbf0f7d39c362a710ac21366723d29b"
server_key_version: "encryptionkey@0"
I20260812 16:38:50.205083 5727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:50.222810 5727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:50.227775 5727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:50.228080 5727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:50.249720 5727 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36585
I20260812 16:38:50.249775 5794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36585 every 8 connection(s)
I20260812 16:38:50.251756 5727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:50.255402 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 5727
I20260812 16:38:50.255643 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:50.256381 5795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:50.256520 31990 external_mini_cluster.cc:1428] Setting key dd30c44dfdb5f57d2825b4f9f88f22ca
I20260812 16:38:50.272472 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: Bootstrap starting.
Aug 12 16:38:50 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:38:50.276578 5795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:50.277778 5795 log.cc:824] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:50.281466 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: No bootstrap required, opened a new log
I20260812 16:38:50.283468 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:50.258826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.31.61.190:36585)
0812 16:38:50.259679 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:50.259695 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:50.259749 (+ 54us) server_negotiation.cc:407] Connection header received
0812 16:38:50.261835 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:50.261870 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:50.262446 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:50.262933 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:50.264313 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:50.266463 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:50.268327 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:50.269554 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:50.269798 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:50.274767 (+ 4969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:50.274779 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:50.274799 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:50.274855 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:50.278431 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:50.279342 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:50.279351 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:50.279359 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:50.279667 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:50.280084 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:50.280089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:50.280092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:50.280837 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:50.281180 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:50.281611 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 16:38:50.282076 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":346,"threads_started":1}
I20260812 16:38:50.292084 5795 raft_consensus.cc:348] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } }
I20260812 16:38:50.292544 5795 raft_consensus.cc:374] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:50.292630 5795 raft_consensus.cc:729] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29b5c9b118c4339b003576414fa9bc9, State: Initialized, Role: FOLLOWER
I20260812 16:38:50.293597 5795 consensus_queue.cc:260] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } }
I20260812 16:38:50.293978 5795 raft_consensus.cc:388] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:50.294191 5795 raft_consensus.cc:482] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:50.294432 5795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:50.297154 5795 raft_consensus.cc:504] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } }
I20260812 16:38:50.297696 5795 leader_election.cc:302] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f29b5c9b118c4339b003576414fa9bc9; no voters:
I20260812 16:38:50.298332 5795 leader_election.cc:288] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:50.298472 5800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:50.299326 5800 raft_consensus.cc:686] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 1 LEADER]: Becoming Leader. State: Replica: f29b5c9b118c4339b003576414fa9bc9, State: Running, Role: LEADER
I20260812 16:38:50.299947 5800 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } }
I20260812 16:38:50.300627 5795 sys_catalog.cc:564] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:50.306411 5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f29b5c9b118c4339b003576414fa9bc9. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } }
I20260812 16:38:50.306818 5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:50.306852 5801 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } }
I20260812 16:38:50.308209 5801 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:50.310213 5815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:50.314908 5815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:50.323945 5815 catalog_manager.cc:1347] Generated new cluster ID: 3201a4a17903469980c8d5870b7a509f
I20260812 16:38:50.324090 5815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:50.346019 5815 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:50.348218 5815 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:50.365218 5815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: Generated new TSK 0
I20260812 16:38:50.366191 5815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab.
Entry for principal oryx/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab.
I20260812 16:38:50.432251 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36585
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab
--principal=oryx/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:50.785950 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:50.786276 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:50.786412 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:50.793876 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:50.794018 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:50.794173 5821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:50.805069 5821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab
--principal=oryx/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:50.807265 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:50.808702 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:50.821707 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:50.821806 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:50.821785 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:50.822885 5821 server_base.cc:1034] running on GCE node
I20260812 16:38:50.823463 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:50.824438 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:50.825647 5821 hybrid_clock.cc:648] HybridClock initialized: now 1786552730825586 us; error 54 us; skew 500 ppm
I20260812 16:38:50.825951 5821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:50 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552730, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:50.828979 5821 init.cc:377] Logged in from keytab as oryx/127.31.61.129@KRBTEST.COM (short username oryx)
I20260812 16:38:50.830402 5821 webserver.cc:460] Webserver started at http://127.31.61.129:44455/ using document root <none> and password file <none>
I20260812 16:38:50.831070 5821 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:50.831171 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:50.831432 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:50.834405 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "21243fc265cc4558b2365fc5d012b97c"
format_stamp: "Formatted at 2026-08-12 16:38:50 on dist-test-slave-f7th"
server_key: "ec1e5237c049286a00bc969b941ee039"
server_key_iv: "789dbd942ba4e05a775d94b3c8557cbe"
server_key_version: "encryptionkey@0"
I20260812 16:38:50.835191 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "21243fc265cc4558b2365fc5d012b97c"
format_stamp: "Formatted at 2026-08-12 16:38:50 on dist-test-slave-f7th"
server_key: "ec1e5237c049286a00bc969b941ee039"
server_key_iv: "789dbd942ba4e05a775d94b3c8557cbe"
server_key_version: "encryptionkey@0"
I20260812 16:38:50.840547 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 16:38:50.844544 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:50.846065 5821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:50.846239 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "21243fc265cc4558b2365fc5d012b97c"
format_stamp: "Formatted at 2026-08-12 16:38:50 on dist-test-slave-f7th"
server_key: "ec1e5237c049286a00bc969b941ee039"
server_key_iv: "789dbd942ba4e05a775d94b3c8557cbe"
server_key_version: "encryptionkey@0"
I20260812 16:38:50.846475 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:50.880154 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:50.885677 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:50.885998 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:50.887064 5821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:50.888531 5821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:50.888614 5821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:50.888732 5821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:50.888790 5821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:50.922402 5821 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:42699
I20260812 16:38:50.922446 5951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:42699 every 8 connection(s)
I20260812 16:38:50.923713 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:50.928483 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 5821
I20260812 16:38:50.928669 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:38:50.929282 31990 external_mini_cluster.cc:1428] Setting key c634781dea6302402a96bcb1be34ca13
Aug 12 16:38:50 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552730, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:38:50.943869 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:50.926372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:37363 (local address 127.31.61.190:36585)
0812 16:38:50.926829 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:50.926834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:50.927697 (+ 863us) server_negotiation.cc:407] Connection header received
0812 16:38:50.929599 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:50.929604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:50.929691 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:50.929789 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:50.931523 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:50.932285 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:50.933812 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:50.934203 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:50.934319 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:50.937938 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:50.937945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:50.937950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:50.937994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:50.940766 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:50.941521 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:50.941526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:50.941529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:50.941674 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:50.942095 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:50.942100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:50.942102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:50.942365 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:50.942497 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:50.943239 (+ 742us) server_negotiation.cc:299] Negotiation successful
0812 16:38:50.943470 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":248,"threads_started":1}
I20260812 16:38:50.944869 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:50.926828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.129:37363)
0812 16:38:50.927520 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:50.927558 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:50.929312 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:50.930036 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:50.930044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:50.930731 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:50.931359 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:50.931377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:50.932436 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:50.932440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:50.933663 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:50.933677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:50.934436 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:50.934443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:50.934674 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:50.935308 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:50.935336 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:50.937686 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:50.940912 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:50.940922 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:50.940941 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:50.941401 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:50.941783 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:50.941788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:50.941792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:50.941988 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:50.942489 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:50.942498 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:50.942852 (+ 354us) client_negotiation.cc:769] Sending connection context
0812 16:38:50.943109 (+ 257us) client_negotiation.cc:240] Negotiation successful
0812 16:38:50.943688 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":297,"threads_started":1}
I20260812 16:38:50.946537 5952 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36585
I20260812 16:38:50.946861 5952 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:50.947685 5952 heartbeater.cc:511] Master 127.31.61.190:36585 requested a full tablet report, sending...
I20260812 16:38:50.949874 5760 ts_manager.cc:194] Registered new tserver with Master: 21243fc265cc4558b2365fc5d012b97c (127.31.61.129:42699)
WARNING: no policy specified for oryx/127.31.61.130@KRBTEST.COM; defaulting to no policy
I20260812 16:38:50.951442 5760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:37363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab.
Entry for principal oryx/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab.
I20260812 16:38:50.983331 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36585
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab
--principal=oryx/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:51.324020 5958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:51.324314 5958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:51.324409 5958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:51.331740 5958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:51.331866 5958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:51.331980 5958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:51.341915 5958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab
--principal=oryx/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:51.343194 5958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:51.344666 5958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:51.359018 5963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:51.359086 5966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:51.359086 5964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:51.359788 5958 server_base.cc:1034] running on GCE node
I20260812 16:38:51.360427 5958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:51.361397 5958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:51.362637 5958 hybrid_clock.cc:648] HybridClock initialized: now 1786552731362582 us; error 61 us; skew 500 ppm
I20260812 16:38:51.362989 5958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:51 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552731, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:51.366359 5958 init.cc:377] Logged in from keytab as oryx/127.31.61.130@KRBTEST.COM (short username oryx)
I20260812 16:38:51.368184 5958 webserver.cc:460] Webserver started at http://127.31.61.130:34883/ using document root <none> and password file <none>
I20260812 16:38:51.368943 5958 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:51.369058 5958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:51.369343 5958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:51.372447 5958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "21211e12dd5f47f49da2ea2d2472e023"
format_stamp: "Formatted at 2026-08-12 16:38:51 on dist-test-slave-f7th"
server_key: "6178b42f7f63746ef89b7fa9a1bdb476"
server_key_iv: "1f94ce0c7fecd92f48a4b39634ce2296"
server_key_version: "encryptionkey@0"
I20260812 16:38:51.373342 5958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "21211e12dd5f47f49da2ea2d2472e023"
format_stamp: "Formatted at 2026-08-12 16:38:51 on dist-test-slave-f7th"
server_key: "6178b42f7f63746ef89b7fa9a1bdb476"
server_key_iv: "1f94ce0c7fecd92f48a4b39634ce2296"
server_key_version: "encryptionkey@0"
I20260812 16:38:51.378830 5958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:38:51.382655 5973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:51.384294 5958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:51.384469 5958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "21211e12dd5f47f49da2ea2d2472e023"
format_stamp: "Formatted at 2026-08-12 16:38:51 on dist-test-slave-f7th"
server_key: "6178b42f7f63746ef89b7fa9a1bdb476"
server_key_iv: "1f94ce0c7fecd92f48a4b39634ce2296"
server_key_version: "encryptionkey@0"
I20260812 16:38:51.384666 5958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:51.410900 5958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:51.415388 5958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:51.415720 5958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:51.416747 5958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:51.418130 5958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:51.418203 5958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:51.418324 5958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:51.418373 5958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:51.452652 5958 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:38581
I20260812 16:38:51.452951 6087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:38581 every 8 connection(s)
I20260812 16:38:51.454072 5958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:51.455861 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 5958
I20260812 16:38:51.456017 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:38:51.456430 31990 external_mini_cluster.cc:1428] Setting key 4b529e0555495e44d2b155838b979e5c
Aug 12 16:38:51 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552731, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.31.61.131@KRBTEST.COM; defaulting to no policy
I20260812 16:38:51.480410 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:51.457423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:43059 (local address 127.31.61.190:36585)
0812 16:38:51.458410 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:51.458416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:51.459118 (+ 702us) server_negotiation.cc:407] Connection header received
0812 16:38:51.461223 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:51.461228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:51.461316 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:51.461431 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:51.463098 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:51.464136 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:51.466151 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:51.466915 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:51.467072 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:51.473222 (+ 6150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:51.473230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:51.473235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:51.473286 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:51.476796 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:51.477730 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:51.477737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:51.477741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:51.477910 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:51.478513 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:51.478518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:51.478527 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:51.478857 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:51.478980 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:51.479883 (+ 903us) server_negotiation.cc:299] Negotiation successful
0812 16:38:51.480100 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":755,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.131@KRBTEST.COM" created.
I20260812 16:38:51.481498 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:51.458014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.130:43059)
0812 16:38:51.458934 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:51.458983 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:51.460897 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:51.461728 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:51.461737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:51.462287 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:51.462903 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:51.462923 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:51.464485 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:51.464492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:51.465922 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:51.465944 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:51.467201 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:51.467207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:51.468465 (+ 1258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:51.469355 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:51.469381 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:51.472859 (+ 3478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:51.477029 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:51.477038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:51.477054 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:51.477583 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:51.478114 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:51.478118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:51.478120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:51.478349 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:51.479016 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:51.479025 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:51.479393 (+ 368us) client_negotiation.cc:769] Sending connection context
0812 16:38:51.479805 (+ 412us) client_negotiation.cc:240] Negotiation successful
0812 16:38:51.480260 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":350,"threads_started":1}
I20260812 16:38:51.483140 6088 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36585
I20260812 16:38:51.483507 6088 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:51.484436 6088 heartbeater.cc:511] Master 127.31.61.190:36585 requested a full tablet report, sending...
I20260812 16:38:51.486138 5760 ts_manager.cc:194] Registered new tserver with Master: 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130:38581)
I20260812 16:38:51.487011 5760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:43059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab.
Entry for principal oryx/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab.
I20260812 16:38:51.507742 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36585
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab
--principal=oryx/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:51.836383 6094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:51.836709 6094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:51.836838 6094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:51.844321 6094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:51.844457 6094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:51.844575 6094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:51.854662 6094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab
--principal=oryx/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:51.855980 6094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:51.857241 6094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:51.870596 6099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:51.870648 6100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:51.870844 6102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:51.871670 6094 server_base.cc:1034] running on GCE node
I20260812 16:38:51.872311 6094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:51.873252 6094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:51.874459 6094 hybrid_clock.cc:648] HybridClock initialized: now 1786552731874413 us; error 35 us; skew 500 ppm
I20260812 16:38:51.874760 6094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:51 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552731, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:51.881400 6094 init.cc:377] Logged in from keytab as oryx/127.31.61.131@KRBTEST.COM (short username oryx)
I20260812 16:38:51.882939 6094 webserver.cc:460] Webserver started at http://127.31.61.131:37627/ using document root <none> and password file <none>
I20260812 16:38:51.883656 6094 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:51.883766 6094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:51.884021 6094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:51.886930 6094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0"
format_stamp: "Formatted at 2026-08-12 16:38:51 on dist-test-slave-f7th"
server_key: "45ff493e84ae33eec8bf1b20a5123f64"
server_key_iv: "beee7d0eb17442e83d659f232c642e52"
server_key_version: "encryptionkey@0"
I20260812 16:38:51.887790 6094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0"
format_stamp: "Formatted at 2026-08-12 16:38:51 on dist-test-slave-f7th"
server_key: "45ff493e84ae33eec8bf1b20a5123f64"
server_key_iv: "beee7d0eb17442e83d659f232c642e52"
server_key_version: "encryptionkey@0"
I20260812 16:38:51.893127 6094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 16:38:51.896696 6109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:51.898041 6094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:38:51.898205 6094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0"
format_stamp: "Formatted at 2026-08-12 16:38:51 on dist-test-slave-f7th"
server_key: "45ff493e84ae33eec8bf1b20a5123f64"
server_key_iv: "beee7d0eb17442e83d659f232c642e52"
server_key_version: "encryptionkey@0"
I20260812 16:38:51.898406 6094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:51.915308 6094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:51.920043 6094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:51.920334 6094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:51.921346 6094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:51.923237 6094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:51.923319 6094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:51.923444 6094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:51.923501 6094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:51.954536 5952 heartbeater.cc:503] Master 127.31.61.190:36585 was elected leader, sending a full tablet report...
I20260812 16:38:51.955147 6094 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:37613
I20260812 16:38:51.955173 6224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:37613 every 8 connection(s)
I20260812 16:38:51.956568 6094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:51.959147 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 6094
I20260812 16:38:51.959311 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:38:51.959760 31990 external_mini_cluster.cc:1428] Setting key 6fd56314ae8419c4e295310a8f38154e
Aug 12 16:38:51 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552731, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:38:51.977106 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:51.959072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:48919 (local address 127.31.61.190:36585)
0812 16:38:51.959274 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:51.959281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:51.960648 (+ 1367us) server_negotiation.cc:407] Connection header received
0812 16:38:51.962945 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:51.962949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:51.963028 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:51.963108 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:51.964750 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:51.965477 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:51.967087 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:51.967503 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:51.967629 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:51.971081 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:51.971085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:51.971089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:51.971121 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:51.973816 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:51.974658 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:51.974663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:51.974666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:51.974811 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:51.975301 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:51.975306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:51.975310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:51.975774 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:51.975929 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:51.976680 (+ 751us) server_negotiation.cc:299] Negotiation successful
0812 16:38:51.976857 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:38:51.978035 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:51.959675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.131:48919)
0812 16:38:51.960471 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:51.960525 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:51.962678 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:51.963315 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:51.963323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:51.963987 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:51.964570 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:51.964588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:51.965651 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:51.965655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:51.966946 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:51.966959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:51.967761 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:51.967766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:51.967969 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:51.968593 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:51.968616 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:51.970863 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:51.973965 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:51.973975 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:51.973989 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:51.974536 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:51.974932 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:51.974937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:51.974941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:51.975178 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:51.975902 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:51.975911 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:51.976282 (+ 371us) client_negotiation.cc:769] Sending connection context
0812 16:38:51.976508 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 16:38:51.976904 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":322,"threads_started":1}
I20260812 16:38:51.979842 6225 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36585
I20260812 16:38:51.980140 6225 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:51.980902 6225 heartbeater.cc:511] Master 127.31.61.190:36585 requested a full tablet report, sending...
I20260812 16:38:51.982470 5760 ts_manager.cc:194] Registered new tserver with Master: c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131:37613)
I20260812 16:38:51.983278 5760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:48919
I20260812 16:38:51.988780 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.61.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.189.keytab.
Entry for principal oryx/127.31.61.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.189.keytab.
I20260812 16:38:52.039495 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.189:43295
--webserver_interface=127.31.61.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.189.keytab
--principal=oryx/127.31.61.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36585,127.31.61.189:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:52.373634 6230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:52.373950 6230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:52.374068 6230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:52.382083 6230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:52.382217 6230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:52.382279 6230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:52.382361 6230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:52.382424 6230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:52.392784 6230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36585,127.31.61.189:43295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.189.keytab
--principal=oryx/127.31.61.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.189:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.61.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
W20260812 16:38:52.393183 6230 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.61.190:36585,127.31.61.189:43295'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 16:38:52.394241 6230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:52.395515 6230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:52.407382 6235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:52.407466 6236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:52.407586 6238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:52.408655 6230 server_base.cc:1034] running on GCE node
I20260812 16:38:52.409559 6230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:52.410996 6230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:52.412297 6230 hybrid_clock.cc:648] HybridClock initialized: now 1786552732412233 us; error 41 us; skew 500 ppm
I20260812 16:38:52.412654 6230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:52 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552732, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:52.415994 6230 init.cc:377] Logged in from keytab as oryx/127.31.61.189@KRBTEST.COM (short username oryx)
I20260812 16:38:52.417475 6230 webserver.cc:460] Webserver started at http://127.31.61.189:35115/ using document root <none> and password file <none>
I20260812 16:38:52.418147 6230 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:52.418234 6230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:52.418512 6230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:52.421546 6230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data/instance:
uuid: "ea9ca77c02c141dc98c05013666e9207"
format_stamp: "Formatted at 2026-08-12 16:38:52 on dist-test-slave-f7th"
server_key: "c01f3bf5ae94cccc907299b32d59857f"
server_key_iv: "80e5cb39e7323ffb443999fd8077e82e"
server_key_version: "encryptionkey@0"
I20260812 16:38:52.422302 6230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal/instance:
uuid: "ea9ca77c02c141dc98c05013666e9207"
format_stamp: "Formatted at 2026-08-12 16:38:52 on dist-test-slave-f7th"
server_key: "c01f3bf5ae94cccc907299b32d59857f"
server_key_iv: "80e5cb39e7323ffb443999fd8077e82e"
server_key_version: "encryptionkey@0"
I20260812 16:38:52.427647 6230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:38:52.431169 6245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:52.432787 6230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 16:38:52.432950 6230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal
uuid: "ea9ca77c02c141dc98c05013666e9207"
format_stamp: "Formatted at 2026-08-12 16:38:52 on dist-test-slave-f7th"
server_key: "c01f3bf5ae94cccc907299b32d59857f"
server_key_iv: "80e5cb39e7323ffb443999fd8077e82e"
server_key_version: "encryptionkey@0"
I20260812 16:38:52.433149 6230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:52.447081 6230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:52.451488 6230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:52.451771 6230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:52.470854 6230 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.189:43295
I20260812 16:38:52.470873 6299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.189:43295 every 8 connection(s)
I20260812 16:38:52.472285 6230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
I20260812 16:38:52.475958 6300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:52.480847 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 6230
I20260812 16:38:52.483311 6300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:38:52.489960 6088 heartbeater.cc:503] Master 127.31.61.190:36585 was elected leader, sending a full tablet report...
Aug 12 16:38:52 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552732, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.189@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
Aug 12 16:38:52 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.189@KRBTEST.COM
I20260812 16:38:52.502796 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.485991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.31.61.190:36585)
0812 16:38:52.486409 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:52.486415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:52.487070 (+ 655us) server_negotiation.cc:407] Connection header received
0812 16:38:52.487143 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:52.487147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:52.487219 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:52.487324 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:52.488613 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.489744 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.491737 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.492300 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.492439 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.495774 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:52.495780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:52.495784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:52.495826 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:52.498678 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.499841 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.499847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.499850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.500009 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.500506 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.500511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.500514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.500937 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:52.501105 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:52.502098 (+ 993us) server_negotiation.cc:299] Negotiation successful
0812 16:38:52.502349 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":222,"threads_started":1}
I20260812 16:38:52.504086 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.483147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.31.61.189:43295)
0812 16:38:52.483950 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:52.483963 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:52.484002 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:38:52.485848 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:52.485881 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:52.486382 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:52.486820 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:52.488413 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.490067 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.491483 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.492113 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.492241 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.496569 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:52.496577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:52.496590 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:52.496619 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:52.499755 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.500499 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.500508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.500511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.500634 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.501107 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.501112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.501115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.501573 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:52.501872 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:52.502291 (+ 419us) server_negotiation.cc:299] Negotiation successful
0812 16:38:52.502714 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":192,"server-negotiator.queue_time_us":594,"spinlock_wait_cycles":22784,"thread_start_us":379,"threads_started":1}
I20260812 16:38:52.504078 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.485992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:41674)
0812 16:38:52.486737 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:52.486778 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:52.486946 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:52.487417 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:52.487423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:52.488040 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:52.488434 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.488450 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.489899 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.489904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.491582 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.491620 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.492513 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.492517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.492645 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.493246 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:52.493268 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:52.495552 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:52.498833 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.498845 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.498860 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.499692 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.500146 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.500149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.500152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.500383 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.501107 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:52.501116 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:52.501615 (+ 499us) client_negotiation.cc:769] Sending connection context
0812 16:38:52.502013 (+ 398us) client_negotiation.cc:240] Negotiation successful
0812 16:38:52.502535 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":419,"threads_started":1}
I20260812 16:38:52.506275 6300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:38:52 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552732, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.189@KRBTEST.COM for oryx/127.31.61.189@KRBTEST.COM
I20260812 16:38:52.519029 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.507246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.0.0.1:56234)
0812 16:38:52.507412 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:52.507431 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:52.507527 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:52.507814 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:52.507817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:52.508132 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:52.508264 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.508271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.509257 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.509261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.510205 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.510216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.510849 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.510853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.510975 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.511498 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:52.511515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:52.513574 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:52.516620 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.516624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.516626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.516992 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.517335 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.517338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.517341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.517466 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.518057 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:52.518063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:52.518335 (+ 272us) client_negotiation.cc:769] Sending connection context
0812 16:38:52.518465 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 16:38:52.518735 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 16:38:52.519224 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.507097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.31.61.189:43295)
0812 16:38:52.507256 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:52.507260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:52.507489 (+ 229us) server_negotiation.cc:407] Connection header received
0812 16:38:52.507632 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:52.507638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:52.507695 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:52.507778 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:52.508372 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.509126 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.510329 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.510735 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.510828 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.513765 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:52.513770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:52.513774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:52.513803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:52.516294 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.517109 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.517114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.517116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.517233 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.517564 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.517567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.517569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.517856 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:52.517979 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:52.518600 (+ 621us) server_negotiation.cc:299] Negotiation successful
0812 16:38:52.518880 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":52}
I20260812 16:38:52.524355 6300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: Bootstrap starting.
I20260812 16:38:52.527516 6300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:52.528616 6300 log.cc:824] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:52.531406 6300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: No bootstrap required, opened a new log
I20260812 16:38:52.538879 6300 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } }
I20260812 16:38:52.539211 6300 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:52.539311 6300 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9ca77c02c141dc98c05013666e9207, State: Initialized, Role: FOLLOWER
I20260812 16:38:52.539929 6300 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } }
I20260812 16:38:52.540766 6307 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } } }
I20260812 16:38:52.541060 6307 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:38:52.541723 6300 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 16:38:52.550077 6318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:38:52.550206 6318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:38:52.554101 6230 master_runner.cc:428] Detected that this master ea9ca77c02c141dc98c05013666e9207 is joining an existing cluster
I20260812 16:38:52.554190 6230 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.61.189:43295
I20260812 16:38:52.580922 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.557925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.31.61.189:43295)
0812 16:38:52.558194 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:52.558203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:52.565238 (+ 7035us) server_negotiation.cc:407] Connection header received
0812 16:38:52.565447 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:52.565452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:52.565532 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:52.565639 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:52.567234 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.568419 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.570501 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.571183 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.571285 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.573971 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:52.573977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:52.573981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:52.574014 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:52.577312 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.578161 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.578166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.578169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.578326 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.578738 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.578742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.578745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.579384 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:52.579521 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:52.580267 (+ 746us) server_negotiation.cc:299] Negotiation successful
0812 16:38:52.580531 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260812 16:38:52.581945 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.558305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.0.0.1:56250)
0812 16:38:52.565110 (+ 6805us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:52.565138 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:52.565324 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:52.565813 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:52.565817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:52.566601 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:52.567057 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.567071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.568675 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.568680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.570347 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.570362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.571639 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.571650 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.571839 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.572552 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:52.572574 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:52.573772 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:52.577438 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.577442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.577446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.578024 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.578469 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.578474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.578477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.578637 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.579633 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:52.579637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:52.579791 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 16:38:52.581214 (+ 1423us) client_negotiation.cc:240] Negotiation successful
0812 16:38:52.581532 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6640,"thread_start_us":6505,"threads_started":1}
WARNING: no policy specified for oryx/127.31.61.188@KRBTEST.COM; defaulting to no policy
I20260812 16:38:52.585112 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.557852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.31.61.190:36585)
0812 16:38:52.558037 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:52.558044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:52.568326 (+ 10282us) server_negotiation.cc:407] Connection header received
0812 16:38:52.568483 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:52.568489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:52.568594 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:52.568688 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:52.570177 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.571203 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.573214 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.573834 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.573924 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.578324 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:52.578331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:52.578335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:52.578378 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:52.581692 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.582905 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.582910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.582913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.583083 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.583525 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.583529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.583532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.583918 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:52.584049 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:52.584544 (+ 495us) server_negotiation.cc:299] Negotiation successful
0812 16:38:52.584780 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 16:38:52.585203 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.558560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:41682)
0812 16:38:52.568182 (+ 9622us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:52.568206 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:52.568349 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:52.568873 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:52.568878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:52.569654 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:52.569991 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.570003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.571416 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.571420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.573025 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.573042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.574228 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.574237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.574412 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.575007 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:52.575035 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:52.578109 (+ 3074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:52.582319 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.582325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.582328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.582768 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.583226 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.583231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.583233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.583400 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.584113 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:52.584116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:52.584280 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 16:38:52.584390 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 16:38:52.584784 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9389,"thread_start_us":9171,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.188@KRBTEST.COM" created.
Aug 12 16:38:52 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552730, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.190@KRBTEST.COM for oryx/127.31.61.189@KRBTEST.COM
I20260812 16:38:52.614979 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.592579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.31.61.189:43295)
0812 16:38:52.592791 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:52.592798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:52.596371 (+ 3573us) server_negotiation.cc:407] Connection header received
0812 16:38:52.596798 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:52.596805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:52.596927 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:52.597060 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:52.598523 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.599749 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.601561 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.602264 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.602383 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.607063 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:52.607071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:52.607075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:52.607118 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:52.611027 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.612492 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.612499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.612503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.612661 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.613198 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.613202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.613204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.613537 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:52.613675 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:52.614327 (+ 652us) server_negotiation.cc:299] Negotiation successful
0812 16:38:52.614571 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 16:38:52.615070 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.592511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.0.0.1:39828)
0812 16:38:52.596228 (+ 3717us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:52.596267 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:52.596543 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:52.597105 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:52.597114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:52.598043 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:52.598318 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.598332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.599885 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.599890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.601393 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.601409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.602731 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.602737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.602918 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.603787 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:52.603812 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:52.606715 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:52.611215 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.611232 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.611241 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.612331 (+ 1090us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.612795 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.612800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.612804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.613050 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.613680 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:52.613690 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:52.613952 (+ 262us) client_negotiation.cc:769] Sending connection context
0812 16:38:52.614188 (+ 236us) client_negotiation.cc:240] Negotiation successful
0812 16:38:52.614580 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3463,"thread_start_us":3284,"threads_started":1}
I20260812 16:38:52.617177 5760 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.61.189:43295: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ea9ca77c02c141dc98c05013666e9207"
member_type: NON_VOTER
last_known_addr {
host: "127.31.61.189"
port: 43295
}
attrs {
promote: true
}
}
dest_uuid: "f29b5c9b118c4339b003576414fa9bc9"
cas_config_opid_index: -1
I20260812 16:38:52.618607 5760 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } }
I20260812 16:38:52.622059 6332 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ea9ca77c02c141dc98c05013666e9207 (127.31.61.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } } }
I20260812 16:38:52.623224 6274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:52.625257 6332 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.61.189:43295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.188.keytab.
Entry for principal oryx/127.31.61.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.188.keytab.
I20260812 16:38:52.627151 6274 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Refusing update from remote peer f29b5c9b118c4339b003576414fa9bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 16:38:52.627444 6230 master.cc:537] Master@127.31.61.189:43295 shutting down...
I20260812 16:38:52.628535 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.188:36255
--webserver_interface=127.31.61.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.188.keytab
--principal=oryx/127.31.61.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36585,127.31.61.189:43295,127.31.61.188:36255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:38:52.628821 6332 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } } }
I20260812 16:38:52.629022 6332 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:52.638329 6331 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:38:52.644577 5744 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 16:38:52.644943 6230 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:52.645385 6230 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:52.645582 6230 tablet_replica.cc:330] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: stopping tablet replica
W20260812 16:38:52.647017 5744 consensus_peers.cc:487] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 -> Peer ea9ca77c02c141dc98c05013666e9207 (127.31.61.189:43295): Couldn't send request to peer ea9ca77c02c141dc98c05013666e9207. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 16:38:52.652889 6230 master.cc:552] Master@127.31.61.189:43295 shutdown complete.
I20260812 16:38:52.653026 6230 master_runner.cc:305] Clearing existing system tablet
I20260812 16:38:52.653434 6230 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:52.653525 6230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:52.658772 6230 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.004s
I20260812 16:38:52.660246 6337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:52.660583 6230 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:52.660846 6230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 16:38:52.660933 6230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal
uuid: "ea9ca77c02c141dc98c05013666e9207"
format_stamp: "Formatted at 2026-08-12 16:38:52 on dist-test-slave-f7th"
server_key: "c01f3bf5ae94cccc907299b32d59857f"
server_key_iv: "80e5cb39e7323ffb443999fd8077e82e"
server_key_version: "encryptionkey@0"
I20260812 16:38:52.664906 6230 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:52.670059 6230 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 16:38:52.670267 6230 log.cc:1196] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 16:38:52.670665 6230 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: Deleting consensus metadata
I20260812 16:38:52.670894 6230 master_runner.cc:315] Copying system tablet from 127.31.61.190:36585
I20260812 16:38:52.673431 6230 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: tablet copy: Beginning tablet copy session from remote peer at address 127.31.61.190:36585
I20260812 16:38:52.689587 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.674382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.31.61.190:36585)
0812 16:38:52.674596 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:52.674604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:52.675026 (+ 422us) server_negotiation.cc:407] Connection header received
0812 16:38:52.675223 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:52.675229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:52.675301 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:52.675382 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:52.676531 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.677686 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.679579 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.680219 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.680315 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.682903 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:52.682911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:52.682915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:52.682955 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:52.686224 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.687087 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.687093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.687096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.687266 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:52.687814 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:52.687819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:52.687823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:52.688191 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:52.688319 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:52.688976 (+ 657us) server_negotiation.cc:299] Negotiation successful
0812 16:38:52.689208 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 16:38:52.690659 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:52.674316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35240)
0812 16:38:52.674923 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:52.674940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:52.675086 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:52.675434 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:52.675439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:52.676040 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:52.676338 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.676351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.677891 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.677897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.679396 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:52.679412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:52.680610 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:52.680616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:52.680794 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:52.681587 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:52.681609 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:52.682666 (+ 1057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:52.686398 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.686403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.686407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.686934 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.687417 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:52.687422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:52.687425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:52.687677 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:52.688418 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:52.688423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:52.688668 (+ 245us) client_negotiation.cc:769] Sending connection context
0812 16:38:52.689873 (+ 1205us) client_negotiation.cc:240] Negotiation successful
0812 16:38:52.690245 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":399,"threads_started":1}
I20260812 16:38:52.691691 5780 tablet_copy_service.cc:140] P f29b5c9b118c4339b003576414fa9bc9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ea9ca77c02c141dc98c05013666e9207 ({username='oryx', principal='oryx/127.31.61.189@KRBTEST.COM'} at 127.0.0.1:35240)
I20260812 16:38:52.692049 5780 tablet_copy_service.cc:161] P f29b5c9b118c4339b003576414fa9bc9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ea9ca77c02c141dc98c05013666e9207 at {username='oryx', principal='oryx/127.31.61.189@KRBTEST.COM'} at 127.0.0.1:35240: session id = ea9ca77c02c141dc98c05013666e9207-00000000000000000000000000000000
I20260812 16:38:52.697075 5780 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 16:38:52.700067 6230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 16:38:52.707940 6230 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: tablet copy: Starting download of 0 data blocks...
I20260812 16:38:52.708632 6230 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: tablet copy: Starting download of 1 WAL segments...
I20260812 16:38:52.712553 6230 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 16:38:52.717423 5780 tablet_copy_service.cc:342] P f29b5c9b118c4339b003576414fa9bc9: Request end of tablet copy session ea9ca77c02c141dc98c05013666e9207-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.61.189@KRBTEST.COM'} at 127.0.0.1:35240
I20260812 16:38:52.717729 5780 tablet_copy_service.cc:433] P f29b5c9b118c4339b003576414fa9bc9: ending tablet copy session ea9ca77c02c141dc98c05013666e9207-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ea9ca77c02c141dc98c05013666e9207
I20260812 16:38:52.730830 6230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:52.733634 6344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:52.734472 6345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:52.742974 6230 server_base.cc:1034] running on GCE node
W20260812 16:38:52.743808 6348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:52.744335 6230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:52.745045 6230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:52.746261 6230 hybrid_clock.cc:648] HybridClock initialized: now 1786552732746204 us; error 32 us; skew 500 ppm
I20260812 16:38:52.746501 6230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:52 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552732, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:52.748404 6230 init.cc:377] Logged in from keytab as oryx/127.31.61.189@KRBTEST.COM (short username oryx)
I20260812 16:38:52.749625 6230 webserver.cc:460] Webserver started at http://127.31.61.189:38949/ using document root <none> and password file <none>
I20260812 16:38:52.749966 6230 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:52.750063 6230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:52.752840 6230 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 16:38:52.755218 6354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:52.755667 6230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 16:38:52.755801 6230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal
uuid: "ea9ca77c02c141dc98c05013666e9207"
format_stamp: "Formatted at 2026-08-12 16:38:52 on dist-test-slave-f7th"
server_key: "c01f3bf5ae94cccc907299b32d59857f"
server_key_iv: "80e5cb39e7323ffb443999fd8077e82e"
server_key_version: "encryptionkey@0"
I20260812 16:38:52.755950 6230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:52.770361 6230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:52.773561 6230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:52.820667 6230 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.189:43295
I20260812 16:38:52.820669 6408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.189:43295 every 8 connection(s)
I20260812 16:38:52.821911 6230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
I20260812 16:38:52.827725 6409 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:38:52.830909 6409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: Bootstrap starting.
I20260812 16:38:52.845160 6409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:38:52.845680 6409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: Bootstrap complete.
I20260812 16:38:52.846835 6409 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } }
I20260812 16:38:52.847041 6409 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea9ca77c02c141dc98c05013666e9207, State: Initialized, Role: LEARNER
I20260812 16:38:52.847258 6409 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } }
I20260812 16:38:52.848436 6409 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:52.848148 6413 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } } }
I20260812 16:38:52.848771 6413 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: This master's current role is: LEARNER
I20260812 16:38:52.858134 6424 catalog_manager.cc:1259] Loaded cluster ID: 3201a4a17903469980c8d5870b7a509f
I20260812 16:38:52.858247 6424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: loading cluster ID for follower catalog manager: success
I20260812 16:38:52.862123 6424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: acquiring CA information for follower catalog manager: success
I20260812 16:38:52.865087 6424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:38:52.986081 6225 heartbeater.cc:503] Master 127.31.61.190:36585 was elected leader, sending a full tablet report...
W20260812 16:38:53.059051 6334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:53.059363 6334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:53.059504 6334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:53.066910 6334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:53.067041 6334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:53.067122 6334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:53.067198 6334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:53.067257 6334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:53.077445 6334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36585,127.31.61.189:43295,127.31.61.188:36255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.188.keytab
--principal=oryx/127.31.61.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.188:36255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.61.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:53.078851 6334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:53.080246 6334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:53.092531 6430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:53.092516 6432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:53.092473 6429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:53.097504 6334 server_base.cc:1034] running on GCE node
I20260812 16:38:53.098402 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:53.100098 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:53.101334 6334 hybrid_clock.cc:648] HybridClock initialized: now 1786552733101289 us; error 37 us; skew 500 ppm
I20260812 16:38:53.101739 6334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:53 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552733, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:53.106032 6334 init.cc:377] Logged in from keytab as oryx/127.31.61.188@KRBTEST.COM (short username oryx)
I20260812 16:38:53.107903 6334 webserver.cc:460] Webserver started at http://127.31.61.188:46333/ using document root <none> and password file <none>
I20260812 16:38:53.108776 6334 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:53.108904 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:53.109283 6334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:53.113587 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data/instance:
uuid: "7d2d7ef1b82d4254a00a1df14193c198"
format_stamp: "Formatted at 2026-08-12 16:38:53 on dist-test-slave-f7th"
server_key: "558762d09ad3b51bc15fad2e6f6908ea"
server_key_iv: "f8af5beb9dc3e107c28498d48fa617e7"
server_key_version: "encryptionkey@0"
I20260812 16:38:53.114528 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal/instance:
uuid: "7d2d7ef1b82d4254a00a1df14193c198"
format_stamp: "Formatted at 2026-08-12 16:38:53 on dist-test-slave-f7th"
server_key: "558762d09ad3b51bc15fad2e6f6908ea"
server_key_iv: "f8af5beb9dc3e107c28498d48fa617e7"
server_key_version: "encryptionkey@0"
I20260812 16:38:53.120476 6334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 16:38:53.124408 6439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:53.125955 6334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:38:53.126149 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal
uuid: "7d2d7ef1b82d4254a00a1df14193c198"
format_stamp: "Formatted at 2026-08-12 16:38:53 on dist-test-slave-f7th"
server_key: "558762d09ad3b51bc15fad2e6f6908ea"
server_key_iv: "f8af5beb9dc3e107c28498d48fa617e7"
server_key_version: "encryptionkey@0"
I20260812 16:38:53.126339 6334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:53.143271 6334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:53.148202 6334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:53.148507 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:53.168727 6334 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.188:36255
I20260812 16:38:53.168752 6490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.188:36255 every 8 connection(s)
I20260812 16:38:53.170120 6334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
I20260812 16:38:53.175567 6491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:53.178293 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 6334
I20260812 16:38:53.185596 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.177393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.0.0.1:39838)
0812 16:38:53.177845 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.177868 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.178021 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.178910 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.178915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.178940 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:53.179116 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.179125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.181166 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.181172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.183106 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.183124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.184812 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.184818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.184997 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.185057 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.185131 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.185275 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260812 16:38:53.186729 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.177537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.31.61.189:43295)
0812 16:38:53.178527 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.178533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.178553 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:38:53.178626 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.178632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.178717 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.178838 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:53.179280 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.180957 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.183340 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.184600 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.185988 (+ 1388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.186201 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.186271 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.186409 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":736,"threads_started":1}
I20260812 16:38:53.186213 6491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:38:53.187871 6382 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260812 16:38:53.188760 6413 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f29b5c9b118c4339b003576414fa9bc9. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } } }
I20260812 16:38:53.189013 6413 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: This master's current role is: LEARNER
Aug 12 16:38:53 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552733, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.188@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
Aug 12 16:38:53 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.188@KRBTEST.COM
I20260812 16:38:53.210186 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.190407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.31.61.190:36585)
0812 16:38:53.190877 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.190884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.191688 (+ 804us) server_negotiation.cc:407] Connection header received
0812 16:38:53.193077 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.193083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.193157 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.193228 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.195091 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.195966 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.198136 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.198805 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.198921 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.203359 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.203367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.203371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.203415 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.206338 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.207313 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.207317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.207319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.207458 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.208031 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.208035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.208038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.208382 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.208568 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.209684 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.209888 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":211,"threads_started":1}
I20260812 16:38:53.211788 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.190493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35252)
0812 16:38:53.191510 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.191566 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.192753 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.193509 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.193517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.194350 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:53.194886 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.194913 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.196107 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.196112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.197993 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.198004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.199058 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.199066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.199461 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.200395 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:53.200421 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:53.203090 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:53.206501 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.206515 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.206529 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.207181 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.207571 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.207575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.207579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.207855 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.208522 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:53.208531 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:53.209198 (+ 667us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.209546 (+ 348us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.210094 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":222,"thread_start_us":562,"threads_started":1}
I20260812 16:38:53.211822 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.180459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.31.61.188:36255)
0812 16:38:53.189843 (+ 9384us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.189854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.190419 (+ 565us) server_negotiation.cc:407] Connection header received
0812 16:38:53.192742 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.192773 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.193087 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.193293 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.195727 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.197347 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.198891 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.199851 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.199998 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.204889 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.204899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.204914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.204948 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.208059 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.209056 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.209064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.209066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.209212 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.209657 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.209661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.209664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.210102 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.210397 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.210915 (+ 518us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.211158 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":9179,"thread_start_us":8963,"threads_started":1}
I20260812 16:38:53.214143 6491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:38:53 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552733, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.188@KRBTEST.COM for oryx/127.31.61.189@KRBTEST.COM
I20260812 16:38:53.227063 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.214896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.0.0.1:39850)
0812 16:38:53.215122 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.215141 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.215224 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.215495 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.215499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.215866 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:53.216023 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.216032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.217040 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.217045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.218083 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.218097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.218842 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.218847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.219002 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.219695 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:53.219714 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:53.221938 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:53.224906 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.224911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.224913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.225311 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.225723 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.225727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.225730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.225867 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.226330 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:53.226333 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:53.226460 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.226601 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.226825 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 16:38:53.227216 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.215040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.31.61.189:43295)
0812 16:38:53.215208 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.215213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.215230 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:38:53.215305 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.215309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.215367 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.215448 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.216158 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.216903 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.218247 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.218701 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.218799 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.222147 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.222153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.222156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.222190 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.224771 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.225429 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.225434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.225436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.225618 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.225944 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.225947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.225949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.226223 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.226335 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.226667 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.226847 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:38:53.228257 6491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:38:53 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552733, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.188@KRBTEST.COM for oryx/127.31.61.188@KRBTEST.COM
I20260812 16:38:53.240540 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.229065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.31.61.188:36255)
0812 16:38:53.229208 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.229213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.229233 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:38:53.229388 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.229393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.229448 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.229525 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.230207 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.230941 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.232176 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.232579 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.232693 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.235787 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.235792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.235795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.235823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.238133 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.238707 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.238712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.238714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.238833 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.239185 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.239190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.239193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.239454 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.239588 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.239958 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.240252 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":58}
I20260812 16:38:53.240538 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.228984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:36255 (local address 127.0.0.1:60642)
0812 16:38:53.229172 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.229192 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.229292 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.229571 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.229576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.229952 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:53.230085 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.230093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.231049 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.231054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.232048 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.232061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.232713 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.232719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.232882 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.233453 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:53.233500 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:53.235563 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:53.238261 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.238267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.238271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.238605 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.238930 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.238935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.238939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.239092 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.239561 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:53.239566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:53.239715 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.239858 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.240113 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 16:38:53.245518 6491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Bootstrap starting.
I20260812 16:38:53.249025 6491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:53.250185 6491 log.cc:824] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:53.253296 6491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: No bootstrap required, opened a new log
I20260812 16:38:53.260771 6491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } }
I20260812 16:38:53.261107 6491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:53.261220 6491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d2d7ef1b82d4254a00a1df14193c198, State: Initialized, Role: FOLLOWER
I20260812 16:38:53.261817 6491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } }
I20260812 16:38:53.262733 6502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } } }
I20260812 16:38:53.263118 6502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:38:53.263727 6491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 16:38:53.272897 6513 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:38:53.273023 6513 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:38:53.277607 6334 master_runner.cc:428] Detected that this master 7d2d7ef1b82d4254a00a1df14193c198 is joining an existing cluster
I20260812 16:38:53.277743 6334 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.61.188:36255
I20260812 16:38:53.303020 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.283709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35264)
0812 16:38:53.284632 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.284649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.284830 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.285363 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.285368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.286304 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:53.287288 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.287304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.288984 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.288990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.290895 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.290910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.292309 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.292316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.292542 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.293406 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:53.293435 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:53.294779 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:53.299513 (+ 4734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.299518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.299522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.300229 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.300698 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.300703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.300706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.300919 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.301695 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:53.301700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:53.301951 (+ 251us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.302139 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.302525 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":112,"spinlock_wait_cycles":1408,"thread_start_us":556,"threads_started":1}
I20260812 16:38:53.303020 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.282350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.31.61.189:43295)
0812 16:38:53.282550 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.282555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.283712 (+ 1157us) server_negotiation.cc:407] Connection header received
0812 16:38:53.283946 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.283951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.284034 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.284143 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.288542 (+ 4399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.289587 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.291838 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.292486 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.292590 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.295207 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.295213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.295218 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.295257 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.299409 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.300365 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.300371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.300375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.300555 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.301083 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.301088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.301106 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.301453 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.301583 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.302303 (+ 720us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.302548 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:38:53.304054 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.282230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.31.61.190:36585)
0812 16:38:53.282400 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.282406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.284731 (+ 2325us) server_negotiation.cc:407] Connection header received
0812 16:38:53.284979 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.284985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.285068 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.285140 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.287502 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.288714 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.291194 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.291912 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.292041 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.295031 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.295038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.295043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.295087 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.299314 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.300378 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.300384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.300387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.300562 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.301055 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.301061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.301065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.301450 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.301618 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.303439 (+ 1821us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.303732 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 16:38:53.304097 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.282376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.0.0.1:39862)
0812 16:38:53.283576 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.283620 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.283811 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.284342 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.284351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.285477 (+ 1126us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:53.288339 (+ 2862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.288353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.289764 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.289769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.291675 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.291691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.292879 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.292885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.293190 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.293801 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:53.293824 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:53.294989 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:53.299574 (+ 4585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.299579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.299583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.300220 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.300707 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.300711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.300715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.300957 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.301726 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:53.301732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:53.302003 (+ 271us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.303278 (+ 1275us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.303588 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":37,"thread_start_us":711,"threads_started":1}
I20260812 16:38:53.305296 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.284803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.31.61.188:36255)
0812 16:38:53.284959 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.284965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.288530 (+ 3565us) server_negotiation.cc:407] Connection header received
0812 16:38:53.288731 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.288736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.288799 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.288857 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.290364 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.291584 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.294315 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.295029 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.295105 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.297313 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.297319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.297323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.297353 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.300794 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.301577 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.301581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.301584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.301735 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.302180 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.302185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.302187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.302777 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.302891 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.304755 (+ 1864us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.304976 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 16:38:53.306470 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.284948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:36255 (local address 127.0.0.1:60650)
0812 16:38:53.288444 (+ 3496us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.288461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.288613 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.289032 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.289036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.289775 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:53.290213 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.290227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.291846 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.291850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.294123 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.294138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.295402 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.295419 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.295584 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.296073 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:53.296095 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:53.297131 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:53.300962 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.300966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.300969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.301452 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.301877 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.301881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.301884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.302082 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.302975 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:53.302979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:53.303873 (+ 894us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.305723 (+ 1850us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.305996 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3294,"mutex_wait_us":7,"thread_start_us":3127,"threads_started":1}
Aug 12 16:38:53 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552730, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.190@KRBTEST.COM for oryx/127.31.61.188@KRBTEST.COM
I20260812 16:38:53.317921 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 5821
I20260812 16:38:53.324945 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.309616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:36255 (local address 127.0.0.1:60652)
0812 16:38:53.309792 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.309810 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.309925 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.310340 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.310345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.310720 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:53.310861 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.310870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.312392 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.312399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.313471 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.313489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.314396 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.314402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.314564 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.315245 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:53.315264 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:53.317669 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:53.322001 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.322007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.322011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.322583 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.323016 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.323022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.323026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.323194 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.323891 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:53.323896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:53.324118 (+ 222us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.324284 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.324592 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 16:38:53.325009 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.309836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.31.61.188:36255)
0812 16:38:53.309986 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.309992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.310029 (+ 37us) server_negotiation.cc:407] Connection header received
0812 16:38:53.310121 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.310124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.310206 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.310281 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.311015 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.312141 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.313673 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.314219 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.314340 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.317898 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.317906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.317911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.317957 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.321840 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.322728 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.322734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.322738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.322892 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.323311 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.323316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.323320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.323752 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.323909 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.324404 (+ 495us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.324641 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 16:38:53.326879 5760 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.61.188:36255: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198"
member_type: NON_VOTER
last_known_addr {
host: "127.31.61.188"
port: 36255
}
attrs {
promote: true
}
}
dest_uuid: "f29b5c9b118c4339b003576414fa9bc9"
cas_config_opid_index: 5
I20260812 16:38:53.328356 5760 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } }
Aug 12 16:38:53 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.129@KRBTEST.COM
I20260812 16:38:53.330346 6493 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7d2d7ef1b82d4254a00a1df14193c198 (127.31.61.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.334188 6493 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.61.188:36255
I20260812 16:38:53.335270 6382 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 LEARNER]: Refusing update from remote peer f29b5c9b118c4339b003576414fa9bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 16:38:53.336894 6466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:53.340802 6466 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Refusing update from remote peer f29b5c9b118c4339b003576414fa9bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 16:38:53.340840 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.318830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.31.61.129:42699)
0812 16:38:53.320129 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.320136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.320161 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:38:53.320218 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.320225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.320445 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.320591 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.321874 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.323369 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.325504 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.326401 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.327136 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.331571 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.331618 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.331644 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.331693 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.335776 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.336834 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.336846 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.336850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.337096 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.337590 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.337595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.337599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.338095 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.339639 (+ 1544us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.339819 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.340097 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1124,"thread_start_us":964,"threads_started":1}
I20260812 16:38:53.341852 6334 master.cc:537] Master@127.31.61.188:36255 shutting down...
I20260812 16:38:53.342599 5947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:53.342296 6523 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 16:38:53.345230 6493 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:53.347667 6524 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.349320 6524 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:53.360375 6382 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7d2d7ef1b82d4254a00a1df14193c198 (127.31.61.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.366220 6413 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: NON_VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: true } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.366583 6413 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: This master's current role is: LEARNER
I20260812 16:38:53.367034 6502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:53.367323 6502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } }
I20260812 16:38:53.368561 6523 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: attempting to promote NON_VOTER ea9ca77c02c141dc98c05013666e9207 to VOTER
I20260812 16:38:53.370009 6523 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } }
I20260812 16:38:53.372678 6382 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 LEARNER]: Refusing update from remote peer f29b5c9b118c4339b003576414fa9bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 16:38:53.373185 6493 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 16:38:53.375325 6525 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ea9ca77c02c141dc98c05013666e9207 (127.31.61.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.376119 6382 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ea9ca77c02c141dc98c05013666e9207 (127.31.61.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.380300 6525 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.380967 6413 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.381278 6413 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:38:53.381933 6502 leader_election.cc:288] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f29b5c9b118c4339b003576414fa9bc9 (127.31.61.190:36585), ea9ca77c02c141dc98c05013666e9207 (127.31.61.189:43295)
I20260812 16:38:53.380429 6523 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.382375 6525 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:53.382386 6523 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:53.383068 6382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d2d7ef1b82d4254a00a1df14193c198" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9ca77c02c141dc98c05013666e9207" is_pre_election: true
I20260812 16:38:53.383589 6334 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:53.384021 6334 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:53.384212 6334 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: stopping tablet replica
W20260812 16:38:53.388463 6443 proxy.cc:239] Call had error, refreshing address and retrying: Service unavailable: reactor is shutting down (error 108)
W20260812 16:38:53.389415 6530 leader_election.cc:334] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f29b5c9b118c4339b003576414fa9bc9 (127.31.61.190:36585): Service unavailable: reactor is shutting down (error 108)
I20260812 16:38:53.389753 6530 leader_election.cc:302] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d2d7ef1b82d4254a00a1df14193c198; no voters: ea9ca77c02c141dc98c05013666e9207, f29b5c9b118c4339b003576414fa9bc9
W20260812 16:38:53.390204 6530 raft_consensus.cc:2670] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: unable to run election callback: Service unavailable: The pool has been shut down.
I20260812 16:38:53.390375 5770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d2d7ef1b82d4254a00a1df14193c198" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29b5c9b118c4339b003576414fa9bc9" is_pre_election: true
W20260812 16:38:53.391064 5746 consensus_peers.cc:487] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 -> Peer 7d2d7ef1b82d4254a00a1df14193c198 (127.31.61.188:36255): Couldn't send request to peer 7d2d7ef1b82d4254a00a1df14193c198. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260812 16:38:53.401629 6334 master.cc:552] Master@127.31.61.188:36255 shutdown complete.
I20260812 16:38:53.401736 6334 master_runner.cc:305] Clearing existing system tablet
I20260812 16:38:53.402082 6334 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:53.402190 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:53.404834 6334 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 16:38:53.406172 6532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:53.406535 6334 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:53.406796 6334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 16:38:53.406889 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal
uuid: "7d2d7ef1b82d4254a00a1df14193c198"
format_stamp: "Formatted at 2026-08-12 16:38:53 on dist-test-slave-f7th"
server_key: "558762d09ad3b51bc15fad2e6f6908ea"
server_key_iv: "f8af5beb9dc3e107c28498d48fa617e7"
server_key_version: "encryptionkey@0"
I20260812 16:38:53.410606 6334 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:53.415324 6334 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 16:38:53.415643 6334 log.cc:1196] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 16:38:53.416141 6334 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Deleting consensus metadata
I20260812 16:38:53.416374 6334 master_runner.cc:315] Copying system tablet from 127.31.61.190:36585
I20260812 16:38:53.418818 6334 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: tablet copy: Beginning tablet copy session from remote peer at address 127.31.61.190:36585
I20260812 16:38:53.431131 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.419627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.31.61.190:36585)
0812 16:38:53.419879 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.419886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.420350 (+ 464us) server_negotiation.cc:407] Connection header received
0812 16:38:53.420554 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.420559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.420630 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.420714 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:53.421575 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.422343 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.423573 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.424073 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.424185 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.425958 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:53.425963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:53.425966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:53.426005 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:53.428471 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.429110 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.429116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.429120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.429252 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:53.429632 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:53.429637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:53.429640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:53.429991 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:53.430130 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.430518 (+ 388us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.430741 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 16:38:53.431180 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.419576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35266)
0812 16:38:53.420243 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.420259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.420407 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.420759 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.420763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.421226 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:53.421425 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.421433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.422497 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.422503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.423434 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.423446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.424225 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.424231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.424417 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.424992 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:53.425012 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:53.425777 (+ 765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:53.428603 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.428609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.428613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.429000 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.429350 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:53.429355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:53.429359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:53.429525 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:53.430108 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:53.430113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:53.430272 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.430395 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.430768 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":380,"threads_started":1}
I20260812 16:38:53.432163 5780 tablet_copy_service.cc:140] P f29b5c9b118c4339b003576414fa9bc9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7d2d7ef1b82d4254a00a1df14193c198 ({username='oryx', principal='oryx/127.31.61.188@KRBTEST.COM'} at 127.0.0.1:35266)
I20260812 16:38:53.432315 5780 tablet_copy_service.cc:161] P f29b5c9b118c4339b003576414fa9bc9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7d2d7ef1b82d4254a00a1df14193c198 at {username='oryx', principal='oryx/127.31.61.188@KRBTEST.COM'} at 127.0.0.1:35266: session id = 7d2d7ef1b82d4254a00a1df14193c198-00000000000000000000000000000000
I20260812 16:38:53.434861 5780 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 16:38:53.436609 6334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 16:38:53.442114 6334 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: tablet copy: Starting download of 0 data blocks...
I20260812 16:38:53.442534 6334 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: tablet copy: Starting download of 1 WAL segments...
I20260812 16:38:53.444787 6334 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 16:38:53.448021 5780 tablet_copy_service.cc:342] P f29b5c9b118c4339b003576414fa9bc9: Request end of tablet copy session 7d2d7ef1b82d4254a00a1df14193c198-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.61.188@KRBTEST.COM'} at 127.0.0.1:35266
I20260812 16:38:53.448148 5780 tablet_copy_service.cc:433] P f29b5c9b118c4339b003576414fa9bc9: ending tablet copy session 7d2d7ef1b82d4254a00a1df14193c198-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7d2d7ef1b82d4254a00a1df14193c198
I20260812 16:38:53.459878 6334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:53.462594 6539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:53.465029 6540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:53.476818 6334 server_base.cc:1034] running on GCE node
W20260812 16:38:53.477160 6542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:53.477633 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:53.478284 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:53.479461 6334 hybrid_clock.cc:648] HybridClock initialized: now 1786552733479416 us; error 35 us; skew 500 ppm
I20260812 16:38:53.479743 6334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:53 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552733, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:53.481479 6334 init.cc:377] Logged in from keytab as oryx/127.31.61.188@KRBTEST.COM (short username oryx)
I20260812 16:38:53.482722 6334 webserver.cc:460] Webserver started at http://127.31.61.188:37793/ using document root <none> and password file <none>
I20260812 16:38:53.482996 6334 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:53.483100 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:53.485507 6334 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 16:38:53.487112 6549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:53.487468 6334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 16:38:53.487614 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal
uuid: "7d2d7ef1b82d4254a00a1df14193c198"
format_stamp: "Formatted at 2026-08-12 16:38:53 on dist-test-slave-f7th"
server_key: "558762d09ad3b51bc15fad2e6f6908ea"
server_key_iv: "f8af5beb9dc3e107c28498d48fa617e7"
server_key_version: "encryptionkey@0"
I20260812 16:38:53.487888 6334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:53.503324 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 5821
I20260812 16:38:53.520871 6334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:53.524127 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:53.526624 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:42699
--local_ip_for_outbound_sockets=127.31.61.129
--tserver_master_addrs=127.31.61.190:36585
--webserver_port=44455
--webserver_interface=127.31.61.129
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab
--principal=oryx/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.61.190:36585,127.31.61.189:43295
--tserver_master_addrs=127.31.61.190:36585,127.31.61.189:43295,127.31.61.188:36255 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:38:53.556733 6334 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.188:36255
I20260812 16:38:53.556751 6602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.188:36255 every 8 connection(s)
I20260812 16:38:53.557746 6334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
I20260812 16:38:53.563531 6603 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:38:53.567888 6603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Bootstrap starting.
I20260812 16:38:53.585202 6603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:38:53.585680 6603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Bootstrap complete.
I20260812 16:38:53.586580 6603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } }
I20260812 16:38:53.586817 6603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d2d7ef1b82d4254a00a1df14193c198, State: Initialized, Role: LEARNER
I20260812 16:38:53.587085 6603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } }
I20260812 16:38:53.587903 6606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.588323 6606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: This master's current role is: LEARNER
I20260812 16:38:53.588374 6603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:53.598230 6617 catalog_manager.cc:1259] Loaded cluster ID: 3201a4a17903469980c8d5870b7a509f
I20260812 16:38:53.598330 6617 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: loading cluster ID for follower catalog manager: success
I20260812 16:38:53.600847 6617 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: acquiring CA information for follower catalog manager: success
I20260812 16:38:53.602766 6617 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:38:53.773962 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.765838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:36255 (local address 127.0.0.1:60654)
0812 16:38:53.766117 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:53.766140 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:53.766300 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:53.767262 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:53.767269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:53.767306 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:53.767533 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.767547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.769480 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.769487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.771427 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:53.771445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.773128 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.773134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:53.773344 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.773381 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:38:53.773479 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 16:38:53.773630 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260812 16:38:53.774076 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:53.766179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.31.61.188:36255)
0812 16:38:53.766868 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:53.766874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:53.766894 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:38:53.766974 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:53.766980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:53.767065 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:53.767180 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:53.767737 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.769299 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.771584 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:53.772936 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:53.773066 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:53.773300 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:53.773526 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 16:38:53.773669 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":372,"threads_started":1}
I20260812 16:38:53.775318 6577 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 16:38:53.776036 6606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f29b5c9b118c4339b003576414fa9bc9. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:53.776338 6606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: This master's current role is: LEARNER
W20260812 16:38:53.918699 6558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:53.919005 6558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:53.919129 6558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:53.926647 6558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:53.926769 6558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:53.926923 6558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:38:53.938102 6558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab
--principal=oryx/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:42699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=44455
--tserver_master_addrs=127.31.61.190:36585,127.31.61.189:43295,127.31.61.188:36255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:53.939488 6558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:53.940828 6558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:53.953945 6623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:53.954034 6624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:53.954208 6626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:53.955391 6558 server_base.cc:1034] running on GCE node
I20260812 16:38:53.957782 6558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:53.958734 6558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:53.959929 6558 hybrid_clock.cc:648] HybridClock initialized: now 1786552733959864 us; error 59 us; skew 500 ppm
I20260812 16:38:53.960227 6558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:53 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552733, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:53.963438 6558 init.cc:377] Logged in from keytab as oryx/127.31.61.129@KRBTEST.COM (short username oryx)
I20260812 16:38:53.965042 6558 webserver.cc:460] Webserver started at http://127.31.61.129:44455/ using document root <none> and password file <none>
I20260812 16:38:53.965714 6558 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:53.965834 6558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:53.971513 6558 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:38:53.974543 6633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:53.975994 6558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:38:53.976147 6558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "21243fc265cc4558b2365fc5d012b97c"
format_stamp: "Formatted at 2026-08-12 16:38:50 on dist-test-slave-f7th"
server_key: "ec1e5237c049286a00bc969b941ee039"
server_key_iv: "789dbd942ba4e05a775d94b3c8557cbe"
server_key_version: "encryptionkey@0"
I20260812 16:38:53.977159 6558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:54.005544 6558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:54.010185 6558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:54.010455 6558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:54.011462 6558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:54.013037 6558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:54.013126 6558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:54.013208 6558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:54.013262 6558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:54.049455 6558 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:42699
I20260812 16:38:54.049557 6748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:42699 every 8 connection(s)
I20260812 16:38:54.050863 6558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:38:54.056424 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 6558
I20260812 16:38:54.056777 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 5958
Aug 12 16:38:54 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552733, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.188@KRBTEST.COM
Aug 12 16:38:54 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552733, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
Aug 12 16:38:54 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552733, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.189@KRBTEST.COM
Aug 12 16:38:54 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.130@KRBTEST.COM
I20260812 16:38:54.078936 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.054204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:47307 (local address 127.31.61.190:36585)
0812 16:38:54.054586 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:54.054592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:54.055393 (+ 801us) server_negotiation.cc:407] Connection header received
0812 16:38:54.059164 (+ 3771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:54.059170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:54.059252 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:54.059328 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:54.062687 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.063741 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.065783 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.066429 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.066524 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.070749 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:54.070756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:54.070760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:54.070802 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:54.074109 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.075798 (+ 1689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.075805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.075809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.075935 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.076612 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.076617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.076621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.077032 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:54.077179 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:54.078313 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0812 16:38:54.078567 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":568,"threads_started":1}
I20260812 16:38:54.080101 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.055979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:33719 (local address 127.31.61.188:36255)
0812 16:38:54.056169 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:54.056176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:54.057743 (+ 1567us) server_negotiation.cc:407] Connection header received
0812 16:38:54.058029 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:54.058036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:54.058116 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:54.058186 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:54.060698 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.061805 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.064660 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.065290 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.065390 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.070630 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:54.070638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:54.070642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:54.070687 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:54.074919 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.077498 (+ 2579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.077505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.077508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.077683 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.078169 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.078175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.078179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.078515 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:54.078649 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:54.079461 (+ 812us) server_negotiation.cc:299] Negotiation successful
0812 16:38:54.079764 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:38:54.080351 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.054281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.129:47307)
0812 16:38:54.055176 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:54.055232 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:54.058982 (+ 3750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:54.059555 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:54.059559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:54.062101 (+ 2542us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:54.062504 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.062517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.063931 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.063936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.065601 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.065618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.066809 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.066815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.067015 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.067575 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:54.067617 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:54.070559 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:54.074839 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.074850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.074872 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.075678 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.076044 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.076049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.076052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.076501 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.077176 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:54.077185 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:54.077824 (+ 639us) client_negotiation.cc:769] Sending connection context
0812 16:38:54.078198 (+ 374us) client_negotiation.cc:240] Negotiation successful
0812 16:38:54.078750 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":2,"spinlock_wait_cycles":88192,"thread_start_us":370,"threads_started":1}
I20260812 16:38:54.080976 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.053768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:35845 (local address 127.31.61.189:43295)
0812 16:38:54.054196 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:54.054202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:54.055346 (+ 1144us) server_negotiation.cc:407] Connection header received
0812 16:38:54.058045 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:54.058050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:54.058140 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:54.058213 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:54.061995 (+ 3782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.063047 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.065970 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.066699 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.066793 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.071539 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:54.071545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:54.071548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:54.071583 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:54.074698 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.077565 (+ 2867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.077571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.077574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.077755 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.079249 (+ 1494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.079254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.079258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.079643 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:54.079782 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:54.080356 (+ 574us) server_negotiation.cc:299] Negotiation successful
0812 16:38:54.080601 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":198,"threads_started":1}
I20260812 16:38:54.081748 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.055810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:36255 (local address 127.31.61.129:33719)
0812 16:38:54.057643 (+ 1833us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:54.057660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:54.057838 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:54.058713 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:54.058725 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:54.060192 (+ 1467us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:54.060539 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.060548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.062016 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.062021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.064443 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.064462 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.065767 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.065774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.066073 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.066877 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:54.066912 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:54.070339 (+ 3427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:54.075030 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.075034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.075037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.077351 (+ 2314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.077834 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.077839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.077842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.078020 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.078727 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:54.078731 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:54.078880 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 16:38:54.080803 (+ 1923us) client_negotiation.cc:240] Negotiation successful
0812 16:38:54.081318 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1719,"mutex_wait_us":68,"thread_start_us":1550,"threads_started":1}
I20260812 16:38:54.082006 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.054598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.31.61.129:35845)
0812 16:38:54.055222 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:54.055240 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:54.057723 (+ 2483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:54.058713 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:54.058725 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:54.059558 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:54.060501 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.060530 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.063284 (+ 2754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.063289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.065823 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.065836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.067064 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.067074 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.067262 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.067781 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:54.067801 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:54.071245 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:54.074839 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.074850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.074872 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.076398 (+ 1526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.077923 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.077928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.077931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.078115 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.079861 (+ 1746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:54.079866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:54.080040 (+ 174us) client_negotiation.cc:769] Sending connection context
0812 16:38:54.081247 (+ 1207us) client_negotiation.cc:240] Negotiation successful
0812 16:38:54.081544 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":4,"spinlock_wait_cycles":4352,"thread_start_us":330,"threads_started":1}
I20260812 16:38:54.082271 6750 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36585
I20260812 16:38:54.082585 6750 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:54.082970 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.058055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.31.61.130:38581)
0812 16:38:54.059469 (+ 1414us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:54.059477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:54.059516 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:38:54.059590 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:54.059622 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:54.059864 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:54.060026 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:54.061989 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.063541 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.065458 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.066297 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.066686 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.075169 (+ 8483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:54.075184 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:54.075199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:54.075235 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:54.078455 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.079571 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.079580 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.079583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.079901 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.080507 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.080513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.080517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.081039 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:54.081446 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:54.082143 (+ 697us) server_negotiation.cc:299] Negotiation successful
0812 16:38:54.082464 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":1082,"threads_started":1}
I20260812 16:38:54.083339 6750 heartbeater.cc:511] Master 127.31.61.190:36585 requested a full tablet report, sending...
I20260812 16:38:54.084892 6751 heartbeater.cc:347] Connected to a master server at 127.31.61.189:43295
I20260812 16:38:54.084951 6083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:54.085013 6751 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:54.085386 6751 heartbeater.cc:511] Master 127.31.61.189:43295 requested a full tablet report, sending...
I20260812 16:38:54.085749 5760 ts_manager.cc:194] Re-registered known tserver with Master: 21243fc265cc4558b2365fc5d012b97c (127.31.61.129:42699)
I20260812 16:38:54.086798 5760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:47307
I20260812 16:38:54.087196 6372 ts_manager.cc:194] Registered new tserver with Master: 21243fc265cc4558b2365fc5d012b97c (127.31.61.129:42699)
I20260812 16:38:54.087719 6749 heartbeater.cc:347] Connected to a master server at 127.31.61.188:36255
I20260812 16:38:54.087859 6749 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:54.088387 6749 heartbeater.cc:511] Master 127.31.61.188:36255 requested a full tablet report, sending...
I20260812 16:38:54.091840 6567 ts_manager.cc:194] Registered new tserver with Master: 21243fc265cc4558b2365fc5d012b97c (127.31.61.129:42699)
I20260812 16:38:54.176250 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 5958
I20260812 16:38:54.191340 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:38581
--local_ip_for_outbound_sockets=127.31.61.130
--tserver_master_addrs=127.31.61.190:36585
--webserver_port=34883
--webserver_interface=127.31.61.130
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab
--principal=oryx/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.61.190:36585,127.31.61.189:43295
--tserver_master_addrs=127.31.61.190:36585,127.31.61.189:43295,127.31.61.188:36255 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:38:54.201018 6523 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: NON_VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: true } } }
I20260812 16:38:54.201550 6523 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:54.203179 6763 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: attempting to promote NON_VOTER 7d2d7ef1b82d4254a00a1df14193c198 to VOTER
I20260812 16:38:54.203838 6763 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } }
I20260812 16:38:54.205708 6382 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Refusing update from remote peer f29b5c9b118c4339b003576414fa9bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 16:38:54.205998 6577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 LEARNER]: Refusing update from remote peer f29b5c9b118c4339b003576414fa9bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 16:38:54.206439 6764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 16:38:54.206869 6763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 16:38:54.209805 6523 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7d2d7ef1b82d4254a00a1df14193c198 (127.31.61.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:54.210325 6382 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7d2d7ef1b82d4254a00a1df14193c198 (127.31.61.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:54.212781 6523 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:54.213012 6523 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:54.213326 6523 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:54.213514 6523 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:54.214536 6764 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:54.214821 6764 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:54.215245 6577 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7d2d7ef1b82d4254a00a1df14193c198 (127.31.61.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:54.218786 6606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:54.219256 6606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:38:54.220709 6767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f29b5c9b118c4339b003576414fa9bc9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:54.221029 6767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 16:38:54.535444 6762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:54.535778 6762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:54.535923 6762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:54.543026 6762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:54.543159 6762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:54.543320 6762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:38:54.554327 6762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab
--principal=oryx/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:38581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=34883
--tserver_master_addrs=127.31.61.190:36585,127.31.61.189:43295,127.31.61.188:36255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:54.555862 6762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:54.557462 6762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:54.572291 6773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:54.572384 6774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:54.572355 6776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:54.573397 6762 server_base.cc:1034] running on GCE node
I20260812 16:38:54.574076 6762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:54.575091 6762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:54.576311 6762 hybrid_clock.cc:648] HybridClock initialized: now 1786552734576265 us; error 31 us; skew 500 ppm
I20260812 16:38:54.576640 6762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:54 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552734, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:54.579962 6762 init.cc:377] Logged in from keytab as oryx/127.31.61.130@KRBTEST.COM (short username oryx)
I20260812 16:38:54.581589 6762 webserver.cc:460] Webserver started at http://127.31.61.130:34883/ using document root <none> and password file <none>
I20260812 16:38:54.582367 6762 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:54.582541 6762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:54.588352 6762 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:38:54.591284 6783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:54.592818 6762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:38:54.592993 6762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "21211e12dd5f47f49da2ea2d2472e023"
format_stamp: "Formatted at 2026-08-12 16:38:51 on dist-test-slave-f7th"
server_key: "6178b42f7f63746ef89b7fa9a1bdb476"
server_key_iv: "1f94ce0c7fecd92f48a4b39634ce2296"
server_key_version: "encryptionkey@0"
I20260812 16:38:54.594000 6762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:54.613281 6762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:54.617430 6762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:54.617697 6762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:54.618685 6762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:54.620244 6762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:54.620342 6762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:54.620456 6762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:54.620527 6762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:54.651664 6762 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:38581
I20260812 16:38:54.651757 6900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:38581 every 8 connection(s)
I20260812 16:38:54.653100 6762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:38:54.654573 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 6762
I20260812 16:38:54.654879 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 6094
Aug 12 16:38:54 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.131@KRBTEST.COM
Aug 12 16:38:54 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552734, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.189@KRBTEST.COM
Aug 12 16:38:54 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552734, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.188@KRBTEST.COM
Aug 12 16:38:54 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552734, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:38:54.693187 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.656278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.31.61.131:37613)
0812 16:38:54.668213 (+ 11935us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:54.668223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:54.669036 (+ 813us) server_negotiation.cc:407] Connection header received
0812 16:38:54.669138 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:54.669148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:54.669438 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:54.669586 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:54.672078 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.674218 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.676857 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.677764 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.677873 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.684612 (+ 6739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:54.684630 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:54.684649 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:54.684694 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:54.688841 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.689762 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.689772 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.689775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.689935 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.690398 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.690403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.690407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.690834 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:54.691154 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:54.692449 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0812 16:38:54.692716 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11709,"thread_start_us":11544,"threads_started":1}
I20260812 16:38:54.694975 6220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:54.695484 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.670512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:55409 (local address 127.31.61.190:36585)
0812 16:38:54.676180 (+ 5668us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:54.676186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:54.676206 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:38:54.676264 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:54.676269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:54.676341 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:54.676449 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:54.677831 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.678834 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.681290 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.681936 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.682070 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.688255 (+ 6185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:54.688260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:54.688263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:54.688298 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:54.691742 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.692764 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.692770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.692774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.692934 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.693494 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.693499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.693503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.693864 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:54.694005 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:54.694886 (+ 881us) server_negotiation.cc:299] Negotiation successful
0812 16:38:54.695107 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5551,"thread_start_us":5451,"threads_started":1}
I20260812 16:38:54.697180 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.656653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:36255 (local address 127.31.61.130:41013)
0812 16:38:54.669325 (+ 12672us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:54.669382 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:54.671885 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:54.675547 (+ 3662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:54.675552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:54.676306 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:54.677095 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.677118 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.678613 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.678618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.681160 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.681170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.682371 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.682379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.682834 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.683671 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:54.683725 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:54.687653 (+ 3928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:54.691758 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.691766 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.691784 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.692578 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.693073 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.693077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.693081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.693360 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.693899 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:54.693908 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:54.694273 (+ 365us) client_negotiation.cc:769] Sending connection context
0812 16:38:54.694501 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 16:38:54.695439 (+ 938us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12352,"mutex_wait_us":228,"spinlock_wait_cycles":50048,"thread_start_us":12119,"threads_started":1}
I20260812 16:38:54.698024 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.670033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.130:55409)
0812 16:38:54.675105 (+ 5072us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:54.675121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:54.675234 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:54.676613 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:54.676618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:54.677173 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:54.677637 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.677651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.679014 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.679019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.681087 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.681122 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.682423 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.682439 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.682816 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.683798 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:54.683826 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:54.687997 (+ 4171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:54.691904 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.691910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.691914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.692611 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.693072 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.693077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.693081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.693355 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.694067 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:54.694071 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:54.694277 (+ 206us) client_negotiation.cc:769] Sending connection context
0812 16:38:54.694505 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 16:38:54.696180 (+ 1675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4959,"mutex_wait_us":1274,"spinlock_wait_cycles":11136,"thread_start_us":5261,"threads_started":1}
I20260812 16:38:54.698566 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.656785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.31.61.130:46303)
0812 16:38:54.673530 (+ 16745us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:54.673551 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:54.673669 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:54.675003 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:54.675017 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:54.676149 (+ 1132us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:54.677535 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.677547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.678959 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.678964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.681076 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:54.681113 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.682371 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.682379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:54.682829 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.683728 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:54.683760 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:54.687986 (+ 4226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:54.694719 (+ 6733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.694724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.694727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.695301 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.696478 (+ 1177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:54.696483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:54.696487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:54.696663 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:54.697448 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:54.697453 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:54.697610 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 16:38:54.697763 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 16:38:54.698122 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16594,"mutex_wait_us":1012,"spinlock_wait_cycles":94208,"thread_start_us":17893,"threads_started":1}
I20260812 16:38:54.699249 6903 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36585
I20260812 16:38:54.699559 6903 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:54.699571 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.656543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:46303 (local address 127.31.61.189:43295)
0812 16:38:54.674164 (+ 17621us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:54.674172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:54.674193 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:38:54.674263 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:54.674268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:54.674346 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:54.674461 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:54.677736 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.678786 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.681284 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.681916 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.682042 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.688955 (+ 6913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:54.688964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:54.688969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:54.689030 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:54.693188 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.696145 (+ 2957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.696151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.696154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.696327 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.696788 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.696793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.696797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.697208 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:54.697363 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:54.698958 (+ 1595us) server_negotiation.cc:299] Negotiation successful
0812 16:38:54.699209 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17482,"thread_start_us":17594,"threads_started":1}
I20260812 16:38:54.700361 6905 heartbeater.cc:347] Connected to a master server at 127.31.61.189:43295
I20260812 16:38:54.700448 6903 heartbeater.cc:511] Master 127.31.61.190:36585 requested a full tablet report, sending...
I20260812 16:38:54.700511 6905 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:54.700775 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:54.657367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:41013 (local address 127.31.61.188:36255)
0812 16:38:54.672648 (+ 15281us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:54.672655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:54.672676 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:38:54.672739 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:54.672745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:54.672826 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:54.672935 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:54.677298 (+ 4363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.678430 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.681309 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:54.681953 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:54.682041 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:54.687969 (+ 5928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:54.687976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:54.687980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:54.688020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:54.691587 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.692758 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.692763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.692767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.692912 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:54.693491 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:54.693496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:54.693500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:54.693770 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:54.693898 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:54.699098 (+ 5200us) server_negotiation.cc:299] Negotiation successful
0812 16:38:54.699335 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15153,"thread_start_us":15009,"threads_started":1}
I20260812 16:38:54.700981 6905 heartbeater.cc:511] Master 127.31.61.189:43295 requested a full tablet report, sending...
I20260812 16:38:54.701913 6901 heartbeater.cc:347] Connected to a master server at 127.31.61.188:36255
I20260812 16:38:54.702045 6901 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:54.702313 5760 ts_manager.cc:194] Re-registered known tserver with Master: 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130:38581)
I20260812 16:38:54.702455 6901 heartbeater.cc:511] Master 127.31.61.188:36255 requested a full tablet report, sending...
I20260812 16:38:54.702553 6372 ts_manager.cc:194] Registered new tserver with Master: 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130:38581)
I20260812 16:38:54.703459 5760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:55409
I20260812 16:38:54.703558 6567 ts_manager.cc:194] Registered new tserver with Master: 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130:38581)
I20260812 16:38:54.786226 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 6094
I20260812 16:38:54.800974 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:37613
--local_ip_for_outbound_sockets=127.31.61.131
--tserver_master_addrs=127.31.61.190:36585
--webserver_port=37627
--webserver_interface=127.31.61.131
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab
--principal=oryx/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.61.190:36585,127.31.61.189:43295
--tserver_master_addrs=127.31.61.190:36585,127.31.61.189:43295,127.31.61.188:36255 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:38:55.089875 6750 heartbeater.cc:503] Master 127.31.61.190:36585 was elected leader, sending a full tablet report...
W20260812 16:38:55.139446 6917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:55.139796 6917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:55.139938 6917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:55.147930 6917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:55.148094 6917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:55.148294 6917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:38:55.161485 6917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab
--principal=oryx/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:37613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=37627
--tserver_master_addrs=127.31.61.190:36585,127.31.61.189:43295,127.31.61.188:36255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:55.163004 6917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:55.164436 6917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:55.178002 6925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:55.177915 6923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:55.178541 6922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:55.179195 6917 server_base.cc:1034] running on GCE node
I20260812 16:38:55.179869 6917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:55.180935 6917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:55.182181 6917 hybrid_clock.cc:648] HybridClock initialized: now 1786552735182109 us; error 57 us; skew 500 ppm
I20260812 16:38:55.182533 6917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:55 dist-test-slave-f7th krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552735, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:55.185865 6917 init.cc:377] Logged in from keytab as oryx/127.31.61.131@KRBTEST.COM (short username oryx)
I20260812 16:38:55.187319 6917 webserver.cc:460] Webserver started at http://127.31.61.131:37627/ using document root <none> and password file <none>
I20260812 16:38:55.188042 6917 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:55.188192 6917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:55.194092 6917 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:38:55.197057 6932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:55.198666 6917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:38:55.198871 6917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0"
format_stamp: "Formatted at 2026-08-12 16:38:51 on dist-test-slave-f7th"
server_key: "45ff493e84ae33eec8bf1b20a5123f64"
server_key_iv: "beee7d0eb17442e83d659f232c642e52"
server_key_version: "encryptionkey@0"
I20260812 16:38:55.199922 6917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:55.212771 6917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:55.217070 6917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:55.217355 6917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:55.218326 6917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:38:55.220036 6917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:38:55.220144 6917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:55.220250 6917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:38:55.220324 6917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:55.253548 6917 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:37613
I20260812 16:38:55.253587 7045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:37613 every 8 connection(s)
I20260812 16:38:55.255138 6917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:38:55.264546 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 6917
Aug 12 16:38:55 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552735, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.189@KRBTEST.COM
Aug 12 16:38:55 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552735, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.188@KRBTEST.COM
Aug 12 16:38:55 dist-test-slave-f7th krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552735, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:38:55.283126 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.259430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:50441 (local address 127.31.61.188:36255)
0812 16:38:55.259938 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.259945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.260010 (+ 65us) server_negotiation.cc:407] Connection header received
0812 16:38:55.262691 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.262698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.262782 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.262925 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.265327 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.266649 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.269724 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.270356 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.270485 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.275864 (+ 5379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.275872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.275877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.275922 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.279321 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.280388 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.280393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.280396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.280544 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.281082 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.281088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.281107 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.281491 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.281660 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.282504 (+ 844us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.282746 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":261,"threads_started":1}
I20260812 16:38:55.284329 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.258871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:36255 (local address 127.31.61.131:50441)
0812 16:38:55.259821 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:55.259891 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:55.262262 (+ 2371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:55.263057 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:55.263069 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:55.264075 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:55.265098 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.265128 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.266885 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.266891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.269530 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.269547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.270610 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.270616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.270915 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.271693 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:55.271727 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:55.275536 (+ 3809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:55.279511 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:55.279522 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:55.279541 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:55.280245 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:55.280691 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:55.280696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:55.280700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:55.280942 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:55.281738 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:55.281747 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:55.282120 (+ 373us) client_negotiation.cc:769] Sending connection context
0812 16:38:55.282378 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 16:38:55.282937 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":360,"spinlock_wait_cycles":32640,"thread_start_us":768,"threads_started":1}
I20260812 16:38:55.285948 7046 heartbeater.cc:347] Connected to a master server at 127.31.61.188:36255
I20260812 16:38:55.286289 7046 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:55.287096 7046 heartbeater.cc:511] Master 127.31.61.188:36255 requested a full tablet report, sending...
I20260812 16:38:55.289083 6567 ts_manager.cc:194] Registered new tserver with Master: c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131:37613)
I20260812 16:38:55.289682 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.258229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:54359 (local address 127.31.61.189:43295)
0812 16:38:55.258695 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.258700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.260335 (+ 1635us) server_negotiation.cc:407] Connection header received
0812 16:38:55.262587 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.262594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.262684 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.262802 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.265315 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.266627 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.269717 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.270356 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.270459 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.276809 (+ 6350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.276817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.276823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.276872 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.281518 (+ 4646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.287004 (+ 5486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.287010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.287013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.287266 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.287752 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.287757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.287760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.288174 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.288311 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.288878 (+ 567us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.289236 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":250,"threads_started":1}
I20260812 16:38:55.290642 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.259520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.31.61.131:54359)
0812 16:38:55.260218 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:55.260238 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:55.262257 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:55.263057 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:55.263069 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:55.264079 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:55.265099 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.265128 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.266911 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.266915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.269530 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.269547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.270770 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.270776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.270947 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.271825 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:55.271848 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:55.276555 (+ 4707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:55.285513 (+ 8958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:55.285519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:55.285523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:55.286088 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:55.287414 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:55.287419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:55.287422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:55.287619 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:55.288390 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:55.288394 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:55.288610 (+ 216us) client_negotiation.cc:769] Sending connection context
0812 16:38:55.290078 (+ 1468us) client_negotiation.cc:240] Negotiation successful
0812 16:38:55.290342 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":338,"spinlock_wait_cycles":8448,"thread_start_us":1311,"threads_started":1}
I20260812 16:38:55.291616 7048 heartbeater.cc:347] Connected to a master server at 127.31.61.189:43295
I20260812 16:38:55.291766 7048 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:55.292265 7048 heartbeater.cc:511] Master 127.31.61.189:43295 requested a full tablet report, sending...
I20260812 16:38:55.292872 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.260712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.131:56153)
0812 16:38:55.261611 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:55.261634 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:55.262280 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:55.263413 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:55.263416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:55.264076 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:38:55.265151 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.265168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.266896 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.266902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.270070 (+ 3168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.270084 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.271366 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.271372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.272161 (+ 789us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.272682 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:38:55.272706 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:38:55.276216 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:38:55.285077 (+ 8861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:55.285082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:55.285085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:55.285631 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:55.289362 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:38:55.289368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:38:55.289372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:38:55.289570 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:38:55.291328 (+ 1758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:38:55.291332 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:38:55.291504 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 16:38:55.291749 (+ 245us) client_negotiation.cc:240] Negotiation successful
0812 16:38:55.292117 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":325,"spinlock_wait_cycles":31488,"thread_start_us":594,"threads_started":1}
I20260812 16:38:55.293049 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.260862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:56153 (local address 127.31.61.190:36585)
0812 16:38:55.261377 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.261384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.261740 (+ 356us) server_negotiation.cc:407] Connection header received
0812 16:38:55.263081 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.263085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.263149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.263240 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.265487 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.266694 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.270237 (+ 3543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.270755 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.270891 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.276701 (+ 5810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.276708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.276713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.276758 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.281706 (+ 4948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.285999 (+ 4293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.286004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.286007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.286249 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.290862 (+ 4613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.290867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.290871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.291218 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.291374 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.292273 (+ 899us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.292526 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":243,"threads_started":1}
I20260812 16:38:55.293777 6372 ts_manager.cc:194] Registered new tserver with Master: c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131:37613)
I20260812 16:38:55.294292 7047 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36585
I20260812 16:38:55.294437 7047 heartbeater.cc:464] Registering TS with master...
I20260812 16:38:55.294932 7047 heartbeater.cc:511] Master 127.31.61.190:36585 requested a full tablet report, sending...
I20260812 16:38:55.296018 5760 ts_manager.cc:194] Re-registered known tserver with Master: c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131:37613)
I20260812 16:38:55.297214 5760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:56153
I20260812 16:38:55.312444 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.271441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.31.61.189:43295)
0812 16:38:55.286982 (+ 15541us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.286988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.287007 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:38:55.287063 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.287069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.287156 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.291693 (+ 4537us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.292765 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.297601 (+ 4836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.300871 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.301720 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.301841 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.305563 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.305570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.305574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.305612 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.308728 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.309620 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.309625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.309629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.309794 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.310242 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.310247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.310250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.310608 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.310836 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.311788 (+ 952us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.312027 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15411,"thread_start_us":17382,"threads_started":1}
I20260812 16:38:55.313591 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.277871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.31.61.190:36585)
0812 16:38:55.293332 (+ 15461us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.293337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.293353 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:38:55.293400 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.293405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.293477 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.293573 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.297002 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.300672 (+ 3670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.302708 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.303372 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.303494 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.307137 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.307144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.307149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.307193 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.310454 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.311301 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.311307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.311311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.311433 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.311842 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.311847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.311851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.312147 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.312364 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.313099 (+ 735us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.313270 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15363,"thread_start_us":21285,"threads_started":1}
I20260812 16:38:55.316820 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.288645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.31.61.188:36255)
0812 16:38:55.288827 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.288837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.299770 (+ 10933us) server_negotiation.cc:407] Connection header received
0812 16:38:55.300111 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.300117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.300252 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.300362 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.303938 (+ 3576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.305044 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.306991 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.307498 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.307642 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.310400 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.310405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.310408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.310442 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.314165 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.314904 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.314909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.314913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.315079 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.315497 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.315503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.315506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.315870 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.316137 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.316407 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.316602 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:38:55.317826 5760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:55.327133 5760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:55.357867 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.346513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.31.61.129:42699)
0812 16:38:55.348062 (+ 1549us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.348071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.348098 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:38:55.348186 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.348192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.348395 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.348563 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:55.349391 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.350379 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.355308 (+ 4929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.356716 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.356857 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.357094 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.357408 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.357581 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":1294,"threads_started":1}
I20260812 16:38:55.357911 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.346210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:42699 (local address 127.0.0.1:35800)
0812 16:38:55.347520 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:55.347543 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:55.347731 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:55.348584 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:55.348594 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:55.348623 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:55.349189 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.349206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.350505 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.350511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.355120 (+ 4609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.355139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.356941 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.356949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.357140 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.357178 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:38:55.357359 (+ 181us) client_negotiation.cc:240] Negotiation successful
0812 16:38:55.357509 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":1008,"threads_started":1}
I20260812 16:38:55.362655 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.347272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.31.61.130:38581)
0812 16:38:55.353286 (+ 6014us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.353298 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.354198 (+ 900us) server_negotiation.cc:407] Connection header received
0812 16:38:55.354335 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.354344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.354645 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.354855 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:55.356154 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.357631 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.360588 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.361597 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.361736 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.361960 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.362197 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.362362 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5819,"thread_start_us":6131,"threads_started":1}
I20260812 16:38:55.362782 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.346689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:38581 (local address 127.0.0.1:43304)
0812 16:38:55.353771 (+ 7082us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:55.353791 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:55.354021 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:55.354917 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:55.354923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:55.354951 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:55.355431 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.355445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.357821 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.357827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.360392 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.360406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.361811 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.361818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.361990 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.362030 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 16:38:55.362130 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 16:38:55.362275 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6940,"mutex_wait_us":59,"spinlock_wait_cycles":768,"thread_start_us":6921,"threads_started":1}
I20260812 16:38:55.362408 6684 tablet_service.cc:1467] Processing CreateTablet for tablet 86ebd9fce84e40abaeaa4a60d98d49d9 (DEFAULT_TABLE table=test-table [id=335257ef21e34eb4a5cc4d62888f730b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:55.363983 6684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ebd9fce84e40abaeaa4a60d98d49d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:55.364034 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.353489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.31.61.131:37613)
0812 16:38:55.355432 (+ 1943us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.355442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.355479 (+ 37us) server_negotiation.cc:407] Connection header received
0812 16:38:55.355588 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.355635 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.355870 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.356076 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:55.356986 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.358465 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.360697 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.362115 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.362233 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.362410 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.363500 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.363717 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":1006,"threads_started":1}
I20260812 16:38:55.366626 6832 tablet_service.cc:1467] Processing CreateTablet for tablet 86ebd9fce84e40abaeaa4a60d98d49d9 (DEFAULT_TABLE table=test-table [id=335257ef21e34eb4a5cc4d62888f730b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:55.368026 6832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ebd9fce84e40abaeaa4a60d98d49d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:55.370503 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.352359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:37613 (local address 127.0.0.1:56116)
0812 16:38:55.354476 (+ 2117us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:55.354498 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:55.354624 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:55.356361 (+ 1737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:55.356365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:55.356393 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:55.356751 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.356763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.358591 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.358597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.360510 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.360525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.362573 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.362584 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.363292 (+ 708us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.363326 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:38:55.363421 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 16:38:55.363546 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1812,"mutex_wait_us":4,"thread_start_us":1633,"threads_started":1}
I20260812 16:38:55.374730 6980 tablet_service.cc:1467] Processing CreateTablet for tablet 86ebd9fce84e40abaeaa4a60d98d49d9 (DEFAULT_TABLE table=test-table [id=335257ef21e34eb4a5cc4d62888f730b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:55.376190 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ebd9fce84e40abaeaa4a60d98d49d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:55.380611 7086 tablet_bootstrap.cc:492] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: Bootstrap starting.
I20260812 16:38:55.384493 7087 tablet_bootstrap.cc:492] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: Bootstrap starting.
I20260812 16:38:55.384704 7086 tablet_bootstrap.cc:654] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:55.387094 7086 log.cc:824] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:55.388923 7087 tablet_bootstrap.cc:654] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:55.390672 7087 log.cc:824] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:55.391535 7089 tablet_bootstrap.cc:492] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Bootstrap starting.
I20260812 16:38:55.394924 7089 tablet_bootstrap.cc:654] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:55.396426 7089 log.cc:824] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:55.397449 7086 tablet_bootstrap.cc:492] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: No bootstrap required, opened a new log
I20260812 16:38:55.397718 7086 ts_tablet_manager.cc:1397] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260812 16:38:55.398356 7087 tablet_bootstrap.cc:492] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: No bootstrap required, opened a new log
I20260812 16:38:55.398672 7087 ts_tablet_manager.cc:1397] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260812 16:38:55.406546 7086 raft_consensus.cc:348] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:55.406804 7089 tablet_bootstrap.cc:492] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: No bootstrap required, opened a new log
I20260812 16:38:55.406924 7086 raft_consensus.cc:374] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:55.407017 7086 raft_consensus.cc:729] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21243fc265cc4558b2365fc5d012b97c, State: Initialized, Role: FOLLOWER
I20260812 16:38:55.407084 7089 ts_tablet_manager.cc:1397] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260812 16:38:55.407936 7086 consensus_queue.cc:260] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:55.409660 7087 raft_consensus.cc:348] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:55.410125 7087 raft_consensus.cc:374] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:55.410225 7087 raft_consensus.cc:729] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21211e12dd5f47f49da2ea2d2472e023, State: Initialized, Role: FOLLOWER
I20260812 16:38:55.411024 7087 consensus_queue.cc:260] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:55.414391 7086 ts_tablet_manager.cc:1428] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: Time spent starting tablet: real 0.017s user 0.000s sys 0.011s
I20260812 16:38:55.418865 6903 heartbeater.cc:503] Master 127.31.61.190:36585 was elected leader, sending a full tablet report...
I20260812 16:38:55.418610 7089 raft_consensus.cc:348] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:55.419090 7089 raft_consensus.cc:374] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:55.419193 7089 raft_consensus.cc:729] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fb7ac1214e4089a3b8c7fa0a0731e0, State: Initialized, Role: FOLLOWER
I20260812 16:38:55.419626 7087 ts_tablet_manager.cc:1428] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: Time spent starting tablet: real 0.021s user 0.013s sys 0.008s
I20260812 16:38:55.419986 7089 consensus_queue.cc:260] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:55.422078 7089 ts_tablet_manager.cc:1428] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260812 16:38:55.422351 7047 heartbeater.cc:503] Master 127.31.61.190:36585 was elected leader, sending a full tablet report...
I20260812 16:38:55.445356 7094 raft_consensus.cc:482] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:55.445672 7094 raft_consensus.cc:504] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:55.447378 7094 leader_election.cc:288] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21243fc265cc4558b2365fc5d012b97c (127.31.61.129:42699), 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130:38581)
I20260812 16:38:55.454010 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.448438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:58157 (local address 127.31.61.130:38581)
0812 16:38:55.448639 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.448645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.448661 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:38:55.448725 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.448730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.448794 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.448867 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:55.449526 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.450300 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.452201 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.453010 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.453162 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.453236 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.453582 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.453741 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:38:55.454023 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.447765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:42699 (local address 127.31.61.131:39809)
0812 16:38:55.448299 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:55.448316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:55.448452 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:55.448886 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:55.448890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:55.448928 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:55.449098 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.449107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.450236 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.450242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.451984 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.451997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.453165 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.453174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.453443 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.453473 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 16:38:55.453560 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 16:38:55.453697 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":226}
I20260812 16:38:55.454066 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.447955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39809 (local address 127.31.61.129:42699)
0812 16:38:55.448155 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.448161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.448399 (+ 238us) server_negotiation.cc:407] Connection header received
0812 16:38:55.448607 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.448613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.448706 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.448794 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:55.449263 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.450076 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.452164 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.452996 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.453155 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.453234 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.453684 (+ 450us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.453831 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:38:55.454075 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.447771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:38581 (local address 127.31.61.131:58157)
0812 16:38:55.447925 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:55.447947 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:55.448096 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:55.448997 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:55.449002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:55.449027 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:55.449371 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.449388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.450437 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.450440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.452007 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:55.452022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.453191 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.453197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:55.453419 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.453452 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:38:55.453533 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 16:38:55.453770 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":119}
I20260812 16:38:55.455236 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ebd9fce84e40abaeaa4a60d98d49d9" candidate_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21243fc265cc4558b2365fc5d012b97c" is_pre_election: true
I20260812 16:38:55.455236 6853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ebd9fce84e40abaeaa4a60d98d49d9" candidate_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21211e12dd5f47f49da2ea2d2472e023" is_pre_election: true
I20260812 16:38:55.455782 6853 raft_consensus.cc:2454] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fb7ac1214e4089a3b8c7fa0a0731e0 in term 0.
I20260812 16:38:55.455792 6704 raft_consensus.cc:2454] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fb7ac1214e4089a3b8c7fa0a0731e0 in term 0.
I20260812 16:38:55.456565 6935 leader_election.cc:302] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21211e12dd5f47f49da2ea2d2472e023, 21243fc265cc4558b2365fc5d012b97c, c1fb7ac1214e4089a3b8c7fa0a0731e0; no voters:
I20260812 16:38:55.457270 7094 raft_consensus.cc:2781] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:55.457420 7094 raft_consensus.cc:482] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:55.457572 7094 raft_consensus.cc:3037] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:55.460539 7094 raft_consensus.cc:504] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:55.461195 7094 leader_election.cc:288] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [CANDIDATE]: Term 1 election: Requested vote from peers 21243fc265cc4558b2365fc5d012b97c (127.31.61.129:42699), 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130:38581)
I20260812 16:38:55.461817 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ebd9fce84e40abaeaa4a60d98d49d9" candidate_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21243fc265cc4558b2365fc5d012b97c"
I20260812 16:38:55.461815 6853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ebd9fce84e40abaeaa4a60d98d49d9" candidate_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21211e12dd5f47f49da2ea2d2472e023"
I20260812 16:38:55.462047 6853 raft_consensus.cc:3037] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:55.462042 6704 raft_consensus.cc:3037] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:55.465350 6853 raft_consensus.cc:2454] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fb7ac1214e4089a3b8c7fa0a0731e0 in term 1.
I20260812 16:38:55.465363 6704 raft_consensus.cc:2454] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fb7ac1214e4089a3b8c7fa0a0731e0 in term 1.
I20260812 16:38:55.465842 6935 leader_election.cc:302] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21211e12dd5f47f49da2ea2d2472e023, 21243fc265cc4558b2365fc5d012b97c, c1fb7ac1214e4089a3b8c7fa0a0731e0; no voters:
I20260812 16:38:55.466147 7094 raft_consensus.cc:2781] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:55.466729 7094 raft_consensus.cc:686] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 1 LEADER]: Becoming Leader. State: Replica: c1fb7ac1214e4089a3b8c7fa0a0731e0, State: Running, Role: LEADER
I20260812 16:38:55.467391 7094 consensus_queue.cc:237] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:55.473884 5759 catalog_manager.cc:5495] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131). New cstate: current_term: 1 leader_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } health_report { overall_health: HEALTHY } } }
W20260812 16:38:55.508562 7050 tablet.cc:2369] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:38:55.515951 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.510645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.31.61.131:37613)
0812 16:38:55.510830 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.510835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.510854 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:38:55.510943 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.510946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.511027 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.511156 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:55.511681 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.512512 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.513887 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.514305 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.514403 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.514882 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:55.515426 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:55.515557 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.515666 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.515747 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:38:55.524991 6853 raft_consensus.cc:1261] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 FOLLOWER]: Refusing update from remote peer c1fb7ac1214e4089a3b8c7fa0a0731e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:55.525092 6704 raft_consensus.cc:1261] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [term 1 FOLLOWER]: Refusing update from remote peer c1fb7ac1214e4089a3b8c7fa0a0731e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:55.526336 7095 consensus_queue.cc:1035] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:55.526695 7094 consensus_queue.cc:1035] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:55.563714 5759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:55.564025 5759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:55.573004 5759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f29b5c9b118c4339b003576414fa9bc9: Sending DeleteTablet for 3 replicas of tablet 86ebd9fce84e40abaeaa4a60d98d49d9
I20260812 16:38:55.574237 6684 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ebd9fce84e40abaeaa4a60d98d49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:55 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:35800
I20260812 16:38:55.574331 6832 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ebd9fce84e40abaeaa4a60d98d49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:55 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:43304
I20260812 16:38:55.575253 7107 tablet_replica.cc:330] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: stopping tablet replica
I20260812 16:38:55.575546 6980 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ebd9fce84e40abaeaa4a60d98d49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:55 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:56116
I20260812 16:38:55.576041 7108 tablet_replica.cc:330] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: stopping tablet replica
I20260812 16:38:55.576825 7107 raft_consensus.cc:2229] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:55.577450 7108 raft_consensus.cc:2229] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:55.578083 7108 raft_consensus.cc:2258] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:55.578630 7107 raft_consensus.cc:2258] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:55.581110 7107 ts_tablet_manager.cc:1905] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:55.581110 7108 ts_tablet_manager.cc:1905] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:55.589787 7107 ts_tablet_manager.cc:1918] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:55.590158 7107 log.cc:1196] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/86ebd9fce84e40abaeaa4a60d98d49d9
I20260812 16:38:55.590588 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 5727
I20260812 16:38:55.591111 7107 ts_tablet_manager.cc:1939] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21211e12dd5f47f49da2ea2d2472e023: Deleting consensus metadata
I20260812 16:38:55.591852 5790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:55.595001 7108 ts_tablet_manager.cc:1918] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:55.595180 7108 log.cc:1196] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/86ebd9fce84e40abaeaa4a60d98d49d9
I20260812 16:38:55.595883 7108 ts_tablet_manager.cc:1939] T 86ebd9fce84e40abaeaa4a60d98d49d9 P 21243fc265cc4558b2365fc5d012b97c: Deleting consensus metadata
I20260812 16:38:55.601900 7109 tablet_replica.cc:330] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: stopping tablet replica
I20260812 16:38:55.602595 7109 raft_consensus.cc:2229] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:55.603559 7109 raft_consensus.cc:2258] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:55.609803 7109 ts_tablet_manager.cc:1905] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:55.613274 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.577157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.31.61.189:43295)
0812 16:38:55.577364 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.577371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.591737 (+ 14366us) server_negotiation.cc:407] Connection header received
0812 16:38:55.592037 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.592042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.592134 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.592218 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.594885 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.596122 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.598515 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.599245 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.599359 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.604697 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.604705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.604711 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.604762 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.609141 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.610525 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.610531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.610535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.610694 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.611275 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.611281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.611284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.611767 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.612062 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.612694 (+ 632us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.612935 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 16:38:55.618492 7109 ts_tablet_manager.cc:1918] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:55.618768 7109 log.cc:1196] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/86ebd9fce84e40abaeaa4a60d98d49d9
I20260812 16:38:55.619654 7109 ts_tablet_manager.cc:1939] T 86ebd9fce84e40abaeaa4a60d98d49d9 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Deleting consensus metadata
I20260812 16:38:55.631698 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.589412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.31.61.188:36255)
0812 16:38:55.589614 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.589620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.615731 (+ 26111us) server_negotiation.cc:407] Connection header received
0812 16:38:55.615989 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.615996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.616080 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.616149 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.618061 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.619225 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.621251 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.621963 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.622104 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.625174 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.625180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.625183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.625267 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.628220 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.629218 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.629222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.629225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.629489 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.629874 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.629878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.629880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.630321 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.630821 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.630958 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.631181 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:38:55.777752 5746 catalog_manager.cc:4841] TS c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131:37613): tablet 86ebd9fce84e40abaeaa4a60d98d49d9 (table test-table [id=335257ef21e34eb4a5cc4d62888f730b]) successfully deleted
I20260812 16:38:55.780398 5743 catalog_manager.cc:4841] TS 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130:38581): tablet 86ebd9fce84e40abaeaa4a60d98d49d9 (table test-table [id=335257ef21e34eb4a5cc4d62888f730b]) successfully deleted
I20260812 16:38:55.780864 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 5727
I20260812 16:38:55.822536 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.806199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.31.61.188:36255)
0812 16:38:55.806454 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.806460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.806973 (+ 513us) server_negotiation.cc:407] Connection header received
0812 16:38:55.807144 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.807150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.807233 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.807324 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.809877 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.810609 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.812302 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.812997 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.813125 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.816819 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.816826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.816831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.816873 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.819336 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.820195 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.820200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.820204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.820354 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.820783 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.820788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.820792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.821181 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.821404 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.821965 (+ 561us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.822195 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 16:38:55.822597 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:55.805232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.31.61.189:43295)
0812 16:38:55.805464 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:55.805471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:55.806424 (+ 953us) server_negotiation.cc:407] Connection header received
0812 16:38:55.807299 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:55.807305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:55.807391 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:55.807504 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:55.809857 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.811008 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.812660 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:55.813130 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:55.813246 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:55.816159 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:55.816163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:55.816167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:55.816201 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:55.819421 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.820409 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.820414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.820417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.820588 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:55.821007 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:55.821012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:55.821015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:55.821373 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:55.821590 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:55.822064 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 16:38:55.822268 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:38:56.549196 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:56.548106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.131:43835)
0812 16:38:56.548671 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:56.548824 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":255,"threads_started":1}
W20260812 16:38:56.549686 7047 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:36585 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
I20260812 16:38:56.558755 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:56.556288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.130:56091)
0812 16:38:56.558361 (+ 2073us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:56.558523 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1957,"thread_start_us":1838,"threads_started":1}
W20260812 16:38:56.559451 6903 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:36585 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
I20260812 16:38:56.560236 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:56.558866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.129:33871)
0812 16:38:56.559983 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:56.560090 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":979,"threads_started":1}
W20260812 16:38:56.560580 6750 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:36585 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
I20260812 16:38:57.072270 7121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f29b5c9b118c4339b003576414fa9bc9)
I20260812 16:38:57.072496 7121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } }
I20260812 16:38:57.073412 7121 leader_election.cc:288] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f29b5c9b118c4339b003576414fa9bc9 (127.31.61.190:36585), ea9ca77c02c141dc98c05013666e9207 (127.31.61.189:43295)
I20260812 16:38:57.085354 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.073679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35720)
0812 16:38:57.084725 (+ 11046us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.084987 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10866,"threads_started":1}
I20260812 16:38:57.086095 7122 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f29b5c9b118c4339b003576414fa9bc9)
I20260812 16:38:57.086393 7122 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } }
I20260812 16:38:57.087400 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.086884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35726)
0812 16:38:57.087095 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.087196 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260812 16:38:57.087631 6553 leader_election.cc:334] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f29b5c9b118c4339b003576414fa9bc9 (127.31.61.190:36585): Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
I20260812 16:38:57.088217 7122 leader_election.cc:288] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f29b5c9b118c4339b003576414fa9bc9 (127.31.61.190:36585), 7d2d7ef1b82d4254a00a1df14193c198 (127.31.61.188:36255)
I20260812 16:38:57.090293 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.088352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35742)
0812 16:38:57.088861 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.089101 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":229,"threads_started":1}
W20260812 16:38:57.090657 6358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
I20260812 16:38:57.092353 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.092001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35756)
0812 16:38:57.092142 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.092213 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 16:38:57.092585 6358 leader_election.cc:334] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f29b5c9b118c4339b003576414fa9bc9 (127.31.61.190:36585): Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
I20260812 16:38:57.093503 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.073709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:43295 (local address 127.0.0.1:39872)
0812 16:38:57.086347 (+ 12638us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.086373 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:57.086564 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:57.086970 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:57.086975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:57.087004 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:57.087312 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.087325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.088844 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.088850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.091090 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.091105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.092740 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.092746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.092908 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.092941 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:38:57.093028 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 16:38:57.093161 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12462,"thread_start_us":11709,"threads_started":1}
I20260812 16:38:57.093561 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.073700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.31.61.189:43295)
0812 16:38:57.086043 (+ 12343us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.086050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.086496 (+ 446us) server_negotiation.cc:407] Connection header received
0812 16:38:57.086744 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.086751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.086815 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.086910 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:57.087498 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.088707 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.091290 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.092602 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.092765 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.092900 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.093064 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.093210 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":12190,"thread_start_us":11955,"threads_started":1}
I20260812 16:38:57.094321 6382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d2d7ef1b82d4254a00a1df14193c198" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea9ca77c02c141dc98c05013666e9207" is_pre_election: true
I20260812 16:38:57.094619 6382 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d2d7ef1b82d4254a00a1df14193c198 in term 1.
I20260812 16:38:57.095129 6551 leader_election.cc:302] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d2d7ef1b82d4254a00a1df14193c198, ea9ca77c02c141dc98c05013666e9207; no voters: f29b5c9b118c4339b003576414fa9bc9
I20260812 16:38:57.095625 7121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 16:38:57.095773 7121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f29b5c9b118c4339b003576414fa9bc9)
I20260812 16:38:57.095897 7121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:38:57.096540 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.089283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.31.61.188:36255)
0812 16:38:57.090195 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.090199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.090215 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:38:57.090257 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.090261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.090309 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.090959 (+ 650us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:57.091325 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.092544 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.094438 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.095546 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.095705 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.095820 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.096118 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.096255 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":669,"threads_started":1}
I20260812 16:38:57.096597 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.088942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:36255 (local address 127.0.0.1:60686)
0812 16:38:57.089607 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.089624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:57.089726 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:57.090468 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:57.090472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:57.090501 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:57.090905 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.090919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.092681 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.092687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.094251 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.094265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.095752 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.095759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.095945 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.095983 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:38:57.096097 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 16:38:57.096233 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":28,"thread_start_us":394,"threads_started":1}
I20260812 16:38:57.097258 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea9ca77c02c141dc98c05013666e9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7d2d7ef1b82d4254a00a1df14193c198" is_pre_election: true
I20260812 16:38:57.098592 7121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } }
I20260812 16:38:57.099225 7121 leader_election.cc:288] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [CANDIDATE]: Term 2 election: Requested vote from peers f29b5c9b118c4339b003576414fa9bc9 (127.31.61.190:36585), ea9ca77c02c141dc98c05013666e9207 (127.31.61.189:43295)
I20260812 16:38:57.099259 6577 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea9ca77c02c141dc98c05013666e9207 in current term 2: Already voted for candidate 7d2d7ef1b82d4254a00a1df14193c198 in this term.
I20260812 16:38:57.099855 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.099380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35770)
0812 16:38:57.099616 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.099737 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260812 16:38:57.099884 6382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d2d7ef1b82d4254a00a1df14193c198" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea9ca77c02c141dc98c05013666e9207"
I20260812 16:38:57.100126 6382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:38:57.100517 6358 leader_election.cc:302] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea9ca77c02c141dc98c05013666e9207; no voters: 7d2d7ef1b82d4254a00a1df14193c198, f29b5c9b118c4339b003576414fa9bc9
I20260812 16:38:57.100919 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.100446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35774)
0812 16:38:57.100619 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.100770 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 16:38:57.101048 6553 leader_election.cc:334] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f29b5c9b118c4339b003576414fa9bc9 (127.31.61.190:36585): Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
I20260812 16:38:57.102671 6382 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d2d7ef1b82d4254a00a1df14193c198 in term 2.
I20260812 16:38:57.102836 7122 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:38:57.103080 6551 leader_election.cc:302] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d2d7ef1b82d4254a00a1df14193c198, ea9ca77c02c141dc98c05013666e9207; no voters: f29b5c9b118c4339b003576414fa9bc9
I20260812 16:38:57.103377 7121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:38:57.103865 7121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [term 2 LEADER]: Becoming Leader. State: Replica: 7d2d7ef1b82d4254a00a1df14193c198, State: Running, Role: LEADER
I20260812 16:38:57.104799 7121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } }
I20260812 16:38:57.106825 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d2d7ef1b82d4254a00a1df14193c198. Latest consensus state: current_term: 2 leader_uuid: "7d2d7ef1b82d4254a00a1df14193c198" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:57.107240 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:57.107981 7133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:57.111954 7133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:57.112771 7133 catalog_manager.cc:1259] Loaded cluster ID: 3201a4a17903469980c8d5870b7a509f
I20260812 16:38:57.112840 7133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:57.114009 7133 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:57.114815 7133 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Loaded TSK: 0
I20260812 16:38:57.115801 7133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:38:57.168972 6567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:38:57.175943 6567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:38:57.180171 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.179748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35804)
0812 16:38:57.179897 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.179988 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 16:38:57.180316 6382 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [term 2 FOLLOWER]: Refusing update from remote peer 7d2d7ef1b82d4254a00a1df14193c198: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260812 16:38:57.181106 7131 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 16:38:57.181265 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.180863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35812)
0812 16:38:57.181046 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.181133 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 16:38:57.181433 6553 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 -> Peer f29b5c9b118c4339b003576414fa9bc9 (127.31.61.190:36585): Couldn't send request to peer f29b5c9b118c4339b003576414fa9bc9. Status: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 16:38:57.183878 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d2d7ef1b82d4254a00a1df14193c198. Latest consensus state: current_term: 2 leader_uuid: "7d2d7ef1b82d4254a00a1df14193c198" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:57.184163 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:38:57.185581 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7d2d7ef1b82d4254a00a1df14193c198" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:57.185925 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198 [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:57.186527 7135 mvcc.cc:205] Tried to move back new op lower bound from 7317720011483394048 to 7317720011185139712. Current Snapshot: MvccSnapshot[applied={T|T < 7317720011483394048}]
I20260812 16:38:57.187125 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7d2d7ef1b82d4254a00a1df14193c198" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f29b5c9b118c4339b003576414fa9bc9" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36585 } } peers { permanent_uuid: "ea9ca77c02c141dc98c05013666e9207" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 43295 } attrs { promote: false } } peers { permanent_uuid: "7d2d7ef1b82d4254a00a1df14193c198" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 36255 } attrs { promote: false } } }
I20260812 16:38:57.187394 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ca77c02c141dc98c05013666e9207 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:38:57.201409 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.194099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:42699 (local address 127.0.0.1:35802)
0812 16:38:57.194813 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.194833 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:57.194948 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:57.195747 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:57.195756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:57.195782 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:57.196090 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.196100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.197186 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.197192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.198745 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.198756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.200698 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.200705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.200872 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.200911 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 16:38:57.200973 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 16:38:57.201149 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"spinlock_wait_cycles":3072}
I20260812 16:38:57.202878 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.194227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.31.61.129:42699)
0812 16:38:57.195360 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.195366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.195385 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:38:57.195469 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.195473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.195562 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.196196 (+ 634us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:57.196323 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.197030 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.198936 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.200387 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.201814 (+ 1427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.201929 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.201983 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.202215 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":930,"threads_started":1}
I20260812 16:38:57.204015 6684 tablet_service.cc:1467] Processing CreateTablet for tablet d77119817a094f1282cde43ed9b06e54 (DEFAULT_TABLE table=test-table [id=21f5f2f326074b45a0453364cfdd127e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:57.204504 6684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77119817a094f1282cde43ed9b06e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:57.205996 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.194634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.31.61.130:38581)
0812 16:38:57.199977 (+ 5343us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.199983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.200012 (+ 29us) server_negotiation.cc:407] Connection header received
0812 16:38:57.200079 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.200085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.200160 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.200259 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:57.200710 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.201780 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.203718 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.204937 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.205079 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.205180 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.205566 (+ 386us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.205709 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5213,"thread_start_us":5119,"threads_started":1}
I20260812 16:38:57.206000 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.194528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:38581 (local address 127.0.0.1:43320)
0812 16:38:57.194817 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.194833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:57.194948 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:57.200320 (+ 5372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:57.200326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:57.200361 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:57.200565 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.200575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.201922 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.201928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.203481 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.203496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.205096 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.205104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.205364 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.205405 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 16:38:57.205513 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 16:38:57.205655 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":24,"spinlock_wait_cycles":4352}
I20260812 16:38:57.208266 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.195037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:37613 (local address 127.0.0.1:56134)
0812 16:38:57.195769 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.195785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:57.195882 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:57.200989 (+ 5107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:57.200994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:57.201022 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:57.201424 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.201435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.203858 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.203864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.205807 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.205822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.207429 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.207435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.207629 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.207665 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 16:38:57.207756 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 16:38:57.207890 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":36,"thread_start_us":457,"threads_started":1}
I20260812 16:38:57.208921 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.195421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.31.61.131:37613)
0812 16:38:57.200703 (+ 5282us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.200710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.200729 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:38:57.200796 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.200801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.200873 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.200973 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:57.201776 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.203693 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.206028 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.207188 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.207332 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.207477 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.208479 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.208631 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5161,"thread_start_us":5082,"threads_started":1}
I20260812 16:38:57.209673 6980 tablet_service.cc:1467] Processing CreateTablet for tablet d77119817a094f1282cde43ed9b06e54 (DEFAULT_TABLE table=test-table [id=21f5f2f326074b45a0453364cfdd127e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:57.210080 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77119817a094f1282cde43ed9b06e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:57.210461 7144 tablet_bootstrap.cc:492] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: Bootstrap starting.
I20260812 16:38:57.210775 6832 tablet_service.cc:1467] Processing CreateTablet for tablet d77119817a094f1282cde43ed9b06e54 (DEFAULT_TABLE table=test-table [id=21f5f2f326074b45a0453364cfdd127e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:38:57.211211 6832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77119817a094f1282cde43ed9b06e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:57.212255 7144 tablet_bootstrap.cc:654] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:57.215766 7146 tablet_bootstrap.cc:492] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Bootstrap starting.
I20260812 16:38:57.217532 7146 tablet_bootstrap.cc:654] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:57.217795 7144 tablet_bootstrap.cc:492] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: No bootstrap required, opened a new log
I20260812 16:38:57.217963 7144 ts_tablet_manager.cc:1397] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260812 16:38:57.218865 7144 raft_consensus.cc:348] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.219187 7144 raft_consensus.cc:374] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:57.219277 7144 raft_consensus.cc:729] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21243fc265cc4558b2365fc5d012b97c, State: Initialized, Role: FOLLOWER
I20260812 16:38:57.219539 7144 consensus_queue.cc:260] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.220501 7144 ts_tablet_manager.cc:1428] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 16:38:57.222131 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.221089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.129:55911)
0812 16:38:57.221712 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.221872 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":309,"threads_started":1}
I20260812 16:38:57.222163 7147 tablet_bootstrap.cc:492] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: Bootstrap starting.
I20260812 16:38:57.224047 7147 tablet_bootstrap.cc:654] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:57.224536 6749 heartbeater.cc:503] Master 127.31.61.188:36255 was elected leader, sending a full tablet report...
I20260812 16:38:57.225086 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.224640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.129:55501)
0812 16:38:57.224855 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.224943 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 16:38:57.226176 7147 tablet_bootstrap.cc:492] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: No bootstrap required, opened a new log
I20260812 16:38:57.226317 7147 ts_tablet_manager.cc:1397] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260812 16:38:57.227063 7147 raft_consensus.cc:348] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.227285 7147 raft_consensus.cc:374] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:57.227375 7147 raft_consensus.cc:729] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21211e12dd5f47f49da2ea2d2472e023, State: Initialized, Role: FOLLOWER
I20260812 16:38:57.227653 7147 consensus_queue.cc:260] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.228430 7147 ts_tablet_manager.cc:1428] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 16:38:57.229811 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.228692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.130:49043)
0812 16:38:57.229361 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.229545 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":425,"threads_started":1}
I20260812 16:38:57.230822 6901 heartbeater.cc:503] Master 127.31.61.188:36255 was elected leader, sending a full tablet report...
I20260812 16:38:57.231681 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.231072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.130:47269)
0812 16:38:57.231418 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.231506 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260812 16:38:57.232751 7146 tablet_bootstrap.cc:492] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: No bootstrap required, opened a new log
I20260812 16:38:57.232882 7146 ts_tablet_manager.cc:1397] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.009s
I20260812 16:38:57.233516 7146 raft_consensus.cc:348] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.233693 7146 raft_consensus.cc:374] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:57.233757 7146 raft_consensus.cc:729] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fb7ac1214e4089a3b8c7fa0a0731e0, State: Initialized, Role: FOLLOWER
I20260812 16:38:57.233947 7146 consensus_queue.cc:260] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.234637 7146 ts_tablet_manager.cc:1428] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 16:38:57.235945 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.234966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.131:59857)
0812 16:38:57.235507 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.235674 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":279,"threads_started":1}
I20260812 16:38:57.237010 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.236564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.131:45363)
0812 16:38:57.236785 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.236868 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"threads_started":1}
I20260812 16:38:57.240267 7046 heartbeater.cc:503] Master 127.31.61.188:36255 was elected leader, sending a full tablet report...
I20260812 16:38:57.272145 7153 raft_consensus.cc:482] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:57.272387 7153 raft_consensus.cc:504] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.273931 7153 leader_election.cc:288] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21243fc265cc4558b2365fc5d012b97c (127.31.61.129:42699), c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131:37613)
I20260812 16:38:57.275815 7149 raft_consensus.cc:482] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:38:57.276022 7149 raft_consensus.cc:504] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.277853 7149 leader_election.cc:288] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130:38581), c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131:37613)
I20260812 16:38:57.281354 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.274364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:47691 (local address 127.31.61.131:37613)
0812 16:38:57.274539 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.274545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.274669 (+ 124us) server_negotiation.cc:407] Connection header received
0812 16:38:57.274941 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.274947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.275027 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.275131 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:57.275669 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.276648 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.278988 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.280190 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.280307 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.280381 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.280920 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.281060 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 16:38:57.282140 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.274229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:37613 (local address 127.31.61.130:47691)
0812 16:38:57.274507 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.274589 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:57.274773 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:57.275163 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:57.275168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:57.275196 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:57.275489 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.275502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.276809 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.276814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.278803 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.278818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.280554 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.280560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.280736 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.280774 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:38:57.281700 (+ 926us) client_negotiation.cc:240] Negotiation successful
0812 16:38:57.281864 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260812 16:38:57.282521 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.274442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:44919 (local address 127.31.61.129:42699)
0812 16:38:57.274619 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.274625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.275138 (+ 513us) server_negotiation.cc:407] Connection header received
0812 16:38:57.275317 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.275322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.275395 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.275490 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:57.276114 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.277183 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.279638 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.280961 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.281136 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.281228 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.282085 (+ 857us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.282239 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:38:57.283196 7001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77119817a094f1282cde43ed9b06e54" candidate_uuid: "21211e12dd5f47f49da2ea2d2472e023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" is_pre_election: true
I20260812 16:38:57.283735 7001 raft_consensus.cc:2454] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21211e12dd5f47f49da2ea2d2472e023 in term 0.
I20260812 16:38:57.284466 6787 leader_election.cc:302] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21211e12dd5f47f49da2ea2d2472e023, c1fb7ac1214e4089a3b8c7fa0a0731e0; no voters:
I20260812 16:38:57.284925 7153 raft_consensus.cc:2781] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:38:57.285043 7153 raft_consensus.cc:482] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:38:57.285133 7153 raft_consensus.cc:3037] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:57.285267 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.278062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:38581 (local address 127.31.61.129:42241)
0812 16:38:57.278421 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.278438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:57.278548 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:57.279202 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:57.279207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:57.279233 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:57.279504 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.279514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.280927 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.280933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.282881 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.282894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.284564 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.284571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.284744 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.284776 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:38:57.284851 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 16:38:57.284997 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260812 16:38:57.285910 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.278732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:42241 (local address 127.31.61.130:38581)
0812 16:38:57.278939 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.278945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.278963 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:38:57.279021 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.279025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.279096 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.279180 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:57.279712 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.280724 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.283059 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.284252 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.284387 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.284456 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.285535 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.285671 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:38:57.286489 6853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77119817a094f1282cde43ed9b06e54" candidate_uuid: "21243fc265cc4558b2365fc5d012b97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21211e12dd5f47f49da2ea2d2472e023" is_pre_election: true
I20260812 16:38:57.287410 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.274327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:42699 (local address 127.31.61.130:44919)
0812 16:38:57.275017 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.275035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:57.275169 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:57.275516 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:57.275519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:57.275539 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:57.275883 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.275895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.277426 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.277431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.279434 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.279448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.281643 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.281653 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.281879 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.281915 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 16:38:57.283255 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0812 16:38:57.286999 (+ 3744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":3569,"thread_start_us":404,"threads_started":1}
I20260812 16:38:57.288064 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77119817a094f1282cde43ed9b06e54" candidate_uuid: "21211e12dd5f47f49da2ea2d2472e023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21243fc265cc4558b2365fc5d012b97c" is_pre_election: true
I20260812 16:38:57.288332 6704 raft_consensus.cc:2454] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21211e12dd5f47f49da2ea2d2472e023 in term 0.
I20260812 16:38:57.288956 7153 raft_consensus.cc:504] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.289481 6853 raft_consensus.cc:2379] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 21243fc265cc4558b2365fc5d012b97c in current term 1: Already voted for candidate 21211e12dd5f47f49da2ea2d2472e023 in this term.
I20260812 16:38:57.289690 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.278642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:56545 (local address 127.31.61.131:37613)
0812 16:38:57.281597 (+ 2955us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.281603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.282818 (+ 1215us) server_negotiation.cc:407] Connection header received
0812 16:38:57.283060 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.283064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.283137 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.283207 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:38:57.283939 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.284933 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.287114 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.288353 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.288487 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.288554 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.289179 (+ 625us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.289306 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2858,"threads_started":1}
I20260812 16:38:57.290585 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.278278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:37613 (local address 127.31.61.129:56545)
0812 16:38:57.282719 (+ 4441us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.282744 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:38:57.282919 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:38:57.283398 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:38:57.283402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:38:57.283439 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:38:57.283776 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.283787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.285082 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.285087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.286929 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:38:57.286944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.288768 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.288774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:38:57.288999 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.289031 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:38:57.289879 (+ 848us) client_negotiation.cc:240] Negotiation successful
0812 16:38:57.290023 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":968,"threads_started":1}
I20260812 16:38:57.290921 7153 leader_election.cc:288] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [CANDIDATE]: Term 1 election: Requested vote from peers 21243fc265cc4558b2365fc5d012b97c (127.31.61.129:42699), c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131:37613)
I20260812 16:38:57.291085 7001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77119817a094f1282cde43ed9b06e54" candidate_uuid: "21211e12dd5f47f49da2ea2d2472e023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0"
I20260812 16:38:57.291191 6704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77119817a094f1282cde43ed9b06e54" candidate_uuid: "21211e12dd5f47f49da2ea2d2472e023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21243fc265cc4558b2365fc5d012b97c"
I20260812 16:38:57.291347 6704 raft_consensus.cc:3037] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:57.291328 7001 raft_consensus.cc:3037] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:57.291361 7000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77119817a094f1282cde43ed9b06e54" candidate_uuid: "21243fc265cc4558b2365fc5d012b97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" is_pre_election: true
I20260812 16:38:57.292413 6637 leader_election.cc:302] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21243fc265cc4558b2365fc5d012b97c; no voters: 21211e12dd5f47f49da2ea2d2472e023, c1fb7ac1214e4089a3b8c7fa0a0731e0
I20260812 16:38:57.293732 6704 raft_consensus.cc:2454] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21211e12dd5f47f49da2ea2d2472e023 in term 1.
I20260812 16:38:57.293922 7149 raft_consensus.cc:2726] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 16:38:57.294103 6786 leader_election.cc:302] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21211e12dd5f47f49da2ea2d2472e023, 21243fc265cc4558b2365fc5d012b97c; no voters:
I20260812 16:38:57.294176 7001 raft_consensus.cc:2454] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21211e12dd5f47f49da2ea2d2472e023 in term 1.
I20260812 16:38:57.294386 7153 raft_consensus.cc:2781] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:57.294874 7153 raft_consensus.cc:686] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 LEADER]: Becoming Leader. State: Replica: 21211e12dd5f47f49da2ea2d2472e023, State: Running, Role: LEADER
I20260812 16:38:57.295297 7153 consensus_queue.cc:237] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } }
I20260812 16:38:57.295935 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.295464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.130:41801)
0812 16:38:57.295662 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.295781 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 16:38:57.299117 6566 catalog_manager.cc:5495] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130). New cstate: current_term: 1 leader_uuid: "21211e12dd5f47f49da2ea2d2472e023" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21211e12dd5f47f49da2ea2d2472e023" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 38581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 } health_report { overall_health: UNKNOWN } } }
W20260812 16:38:57.307204 6752 tablet.cc:2369] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:38:57.333487 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.328038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.31.61.130:38581)
0812 16:38:57.328273 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.328280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.328300 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:38:57.328476 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.328480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.328569 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.328705 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:38:57.329073 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.329814 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.331272 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.331777 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.331895 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.332307 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:38:57.332862 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:38:57.332994 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.333102 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.333187 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260812 16:38:57.337594 6704 raft_consensus.cc:1261] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 1 FOLLOWER]: Refusing update from remote peer 21211e12dd5f47f49da2ea2d2472e023: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:57.337772 7001 raft_consensus.cc:1261] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 1 FOLLOWER]: Refusing update from remote peer 21211e12dd5f47f49da2ea2d2472e023: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:38:57.338266 7161 consensus_queue.cc:1035] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21243fc265cc4558b2365fc5d012b97c" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:57.338620 7153 consensus_queue.cc:1035] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1fb7ac1214e4089a3b8c7fa0a0731e0" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:38:57.341703 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.340925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.131:36185)
0812 16:38:57.341233 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.341437 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260812 16:38:57.342022 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.341073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.129:50285)
0812 16:38:57.341549 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.341648 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
I20260812 16:38:57.347311 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.346725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.129:50891)
0812 16:38:57.347034 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.347134 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260812 16:38:57.348619 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.347877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.130:44431)
0812 16:38:57.348346 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.348450 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360}
I20260812 16:38:57.351698 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.351292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.130:54373)
0812 16:38:57.351415 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.351531 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 16:38:57.352952 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.352498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.31.61.131:38531)
0812 16:38:57.352710 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.352798 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260812 16:38:57.363897 6567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:57.364212 6567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:38:57.370447 6567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d2d7ef1b82d4254a00a1df14193c198: Sending DeleteTablet for 3 replicas of tablet d77119817a094f1282cde43ed9b06e54
I20260812 16:38:57.371557 6684 tablet_service.cc:1514] Processing DeleteTablet for tablet d77119817a094f1282cde43ed9b06e54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:57 UTC) from {username='oryx', principal='oryx/127.31.61.188@KRBTEST.COM'} at 127.0.0.1:35802
I20260812 16:38:57.372318 6832 tablet_service.cc:1514] Processing DeleteTablet for tablet d77119817a094f1282cde43ed9b06e54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:57 UTC) from {username='oryx', principal='oryx/127.31.61.188@KRBTEST.COM'} at 127.0.0.1:43320
I20260812 16:38:57.372757 6980 tablet_service.cc:1514] Processing DeleteTablet for tablet d77119817a094f1282cde43ed9b06e54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:38:57 UTC) from {username='oryx', principal='oryx/127.31.61.188@KRBTEST.COM'} at 127.0.0.1:56134
I20260812 16:38:57.374681 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 6558
I20260812 16:38:57.383682 7174 tablet_replica.cc:330] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: stopping tablet replica
I20260812 16:38:57.384102 7174 raft_consensus.cc:2229] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:38:57.384931 7174 raft_consensus.cc:2258] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:57.386318 7174 ts_tablet_manager.cc:1905] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:57.391690 7173 tablet_replica.cc:330] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: stopping tablet replica
I20260812 16:38:57.392050 7173 raft_consensus.cc:2229] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:57.392408 7173 raft_consensus.cc:2258] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:57.393308 7175 tablet_replica.cc:330] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: stopping tablet replica
I20260812 16:38:57.393590 7175 raft_consensus.cc:2229] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:38:57.393743 7173 ts_tablet_manager.cc:1905] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:57.393935 7175 raft_consensus.cc:2258] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:38:57.394929 7175 ts_tablet_manager.cc:1905] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:38:57.394944 7174 ts_tablet_manager.cc:1918] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:57.395212 7174 log.cc:1196] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/d77119817a094f1282cde43ed9b06e54
I20260812 16:38:57.395810 7174 ts_tablet_manager.cc:1939] T d77119817a094f1282cde43ed9b06e54 P 21211e12dd5f47f49da2ea2d2472e023: Deleting consensus metadata
I20260812 16:38:57.397151 6550 catalog_manager.cc:4841] TS 21211e12dd5f47f49da2ea2d2472e023 (127.31.61.130:38581): tablet d77119817a094f1282cde43ed9b06e54 (table test-table [id=21f5f2f326074b45a0453364cfdd127e]) successfully deleted
I20260812 16:38:57.400766 7175 ts_tablet_manager.cc:1918] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:57.400933 7175 log.cc:1196] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/d77119817a094f1282cde43ed9b06e54
I20260812 16:38:57.401475 7175 ts_tablet_manager.cc:1939] T d77119817a094f1282cde43ed9b06e54 P c1fb7ac1214e4089a3b8c7fa0a0731e0: Deleting consensus metadata
I20260812 16:38:57.402271 7173 ts_tablet_manager.cc:1918] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:38:57.402415 7173 log.cc:1196] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/d77119817a094f1282cde43ed9b06e54
I20260812 16:38:57.402863 7173 ts_tablet_manager.cc:1939] T d77119817a094f1282cde43ed9b06e54 P 21243fc265cc4558b2365fc5d012b97c: Deleting consensus metadata
I20260812 16:38:57.402971 6553 catalog_manager.cc:4841] TS c1fb7ac1214e4089a3b8c7fa0a0731e0 (127.31.61.131:37613): tablet d77119817a094f1282cde43ed9b06e54 (table test-table [id=21f5f2f326074b45a0453364cfdd127e]) successfully deleted
I20260812 16:38:57.405222 6552 catalog_manager.cc:4841] TS 21243fc265cc4558b2365fc5d012b97c (127.31.61.129:42699): tablet d77119817a094f1282cde43ed9b06e54 (table test-table [id=21f5f2f326074b45a0453364cfdd127e]) successfully deleted
I20260812 16:38:57.413952 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.375777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.31.61.129:42699)
0812 16:38:57.375968 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.375974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.389095 (+ 13121us) server_negotiation.cc:407] Connection header received
0812 16:38:57.389938 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.389945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.390043 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.390130 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:57.399799 (+ 9669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.400915 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.403213 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.403893 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.404054 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.406767 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:57.406786 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:57.406808 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:57.406859 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:57.410610 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:57.411288 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:57.411298 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:57.411302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:57.411466 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:57.412001 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:57.412006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:57.412010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:57.412542 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:57.413082 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.413288 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.413518 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:38:57.414834 6744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:57.533002 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 6558
I20260812 16:38:57.549223 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 6762
I20260812 16:38:57.567788 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.550053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.31.61.130:38581)
0812 16:38:57.550254 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.550258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.550276 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:38:57.550465 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.550469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.550534 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.550652 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:57.552376 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.553575 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.555345 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.556202 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.556329 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.559692 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:57.559712 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:57.559733 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:57.559804 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:57.564135 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:57.565093 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:57.565105 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:57.565124 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:57.565311 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:57.565768 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:57.565774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:57.565778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:57.566320 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:57.566738 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.566951 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.567214 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 16:38:57.568661 6896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:57.679697 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.679123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35822)
0812 16:38:57.679335 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.679461 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 16:38:57.680780 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.680380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36585 (local address 127.0.0.1:35838)
0812 16:38:57.680568 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 16:38:57.680665 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 16:38:57.686698 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 6762
I20260812 16:38:57.703534 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 6917
I20260812 16:38:57.717165 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:57.704326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.31.61.131:37613)
0812 16:38:57.704485 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:57.704490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:57.704556 (+ 66us) server_negotiation.cc:407] Connection header received
0812 16:38:57.704747 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:57.704753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:57.704838 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:57.704925 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:57.706357 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.707108 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.708472 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:57.708901 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:57.709005 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:57.711547 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:57.711562 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:57.711580 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:57.711648 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:57.714419 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:57.715147 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:57.715155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:57.715158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:57.715306 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:57.715753 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:57.715756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:57.715758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:57.716160 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:57.716384 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:57.716731 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 16:38:57.716909 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 16:38:57.717856 7041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:57.839097 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 6917
I20260812 16:38:57.857683 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 6230
I20260812 16:38:57.858820 6404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:57.977725 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 6230
I20260812 16:38:57.996066 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 6334
I20260812 16:38:57.997119 6598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:38:58.121698 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 6334
2026-08-12T16:38:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9629 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:38:58 dist-test-slave-f7th krb5kdc[7185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:38:58 dist-test-slave-f7th krb5kdc[7185](info): set up 2 sockets
Aug 12 16:38:58 dist-test-slave-f7th krb5kdc[7185](info): commencing operation
krb5kdc: starting...
W20260812 16:38:59.164436 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:38:59 dist-test-slave-f7th krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:38:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.190.keytab.
Entry for principal oryx/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.190.keytab.
I20260812 16:38:59.397604 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:38115
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.190.keytab
--principal=oryx/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:38115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:38:59.742055 7198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:38:59.742350 7198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:38:59.742498 7198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:38:59.750020 7198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:38:59.750150 7198 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 16:38:59.750226 7198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:38:59.750316 7198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:38:59.750381 7198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:38:59.750461 7198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:38:59.760532 7198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:38115
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.190.keytab
--principal=oryx/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:38115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:38:59.761842 7198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:38:59.763190 7198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:38:59.775029 7204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:59.775089 7207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:38:59.775118 7205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:38:59.775817 7198 server_base.cc:1034] running on GCE node
I20260812 16:38:59.776564 7198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:38:59.777979 7198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:38:59.779233 7198 hybrid_clock.cc:648] HybridClock initialized: now 1786552739779171 us; error 34 us; skew 500 ppm
I20260812 16:38:59.779565 7198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:38:59 dist-test-slave-f7th krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552739, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:38:59.782819 7198 init.cc:377] Logged in from keytab as oryx/127.31.61.190@KRBTEST.COM (short username oryx)
I20260812 16:38:59.784413 7198 webserver.cc:460] Webserver started at http://127.31.61.190:42065/ using document root <none> and password file <none>
I20260812 16:38:59.785116 7198 fs_manager.cc:359] Metadata directory not provided
I20260812 16:38:59.785220 7198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:38:59.785532 7198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:38:59.788532 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "e1e4436585474a36b26f7ac87802a0ce"
format_stamp: "Formatted at 2026-08-12 16:38:59 on dist-test-slave-f7th"
server_key: "099ab6dd856476a5e6ed1192ba3a6e52"
server_key_iv: "40f35a108cfbb33967a1686ab6a7b053"
server_key_version: "encryptionkey@0"
I20260812 16:38:59.789307 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "e1e4436585474a36b26f7ac87802a0ce"
format_stamp: "Formatted at 2026-08-12 16:38:59 on dist-test-slave-f7th"
server_key: "099ab6dd856476a5e6ed1192ba3a6e52"
server_key_iv: "40f35a108cfbb33967a1686ab6a7b053"
server_key_version: "encryptionkey@0"
I20260812 16:38:59.794389 7198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:38:59.797784 7214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:38:59.799060 7198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:38:59.799223 7198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "e1e4436585474a36b26f7ac87802a0ce"
format_stamp: "Formatted at 2026-08-12 16:38:59 on dist-test-slave-f7th"
server_key: "099ab6dd856476a5e6ed1192ba3a6e52"
server_key_iv: "40f35a108cfbb33967a1686ab6a7b053"
server_key_version: "encryptionkey@0"
I20260812 16:38:59.799419 7198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:38:59.813459 7198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:38:59.817907 7198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:38:59.818189 7198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:38:59.839479 7198 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:38115
I20260812 16:38:59.839502 7275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:38115 every 8 connection(s)
I20260812 16:38:59.840899 7198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:38:59.844637 7276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:38:59.850677 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 7198
I20260812 16:38:59.850900 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:38:59.851743 31990 external_mini_cluster.cc:1428] Setting key 23b09cf7af4e5c8fccc73bb890104478
I20260812 16:38:59.857024 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce: Bootstrap starting.
I20260812 16:38:59.861107 7276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce: Neither blocks nor log segments found. Creating new log.
I20260812 16:38:59.862470 7276 log.cc:824] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce: Log is configured to *not* fsync() on all Append() calls
I20260812 16:38:59.866562 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce: No bootstrap required, opened a new log
Aug 12 16:38:59 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:38:59.875953 7276 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4436585474a36b26f7ac87802a0ce" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38115 } }
I20260812 16:38:59.876507 7276 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:38:59.876617 7276 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e4436585474a36b26f7ac87802a0ce, State: Initialized, Role: FOLLOWER
I20260812 16:38:59.877120 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:38:59.854194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.31.61.190:38115)
0812 16:38:59.855082 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0812 16:38:59.855095 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:38:59.855138 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:38:59.856961 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:38:59.856987 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:38:59.857418 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:38:59.857777 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:38:59.859767 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:59.861200 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:59.863044 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:38:59.864155 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:38:59.864339 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:38:59.869123 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:38:59.869133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:38:59.869149 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:38:59.869195 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:38:59.872592 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:59.873350 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:59.873357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:59.873364 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:59.873582 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:38:59.873969 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:38:59.873973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:38:59.873975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:38:59.874618 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:38:59.874926 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 16:38:59.875470 (+ 544us) server_negotiation.cc:299] Negotiation successful
0812 16:38:59.875957 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":438,"threads_started":1}
I20260812 16:38:59.877228 7276 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4436585474a36b26f7ac87802a0ce" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38115 } }
I20260812 16:38:59.877513 7276 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:38:59.877805 7276 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:38:59.878041 7276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:38:59.880707 7276 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4436585474a36b26f7ac87802a0ce" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38115 } }
I20260812 16:38:59.881246 7276 leader_election.cc:302] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e4436585474a36b26f7ac87802a0ce; no voters:
I20260812 16:38:59.881876 7276 leader_election.cc:288] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:38:59.882012 7281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:38:59.882649 7281 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [term 1 LEADER]: Becoming Leader. State: Replica: e1e4436585474a36b26f7ac87802a0ce, State: Running, Role: LEADER
I20260812 16:38:59.883234 7281 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4436585474a36b26f7ac87802a0ce" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38115 } }
I20260812 16:38:59.884143 7276 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:38:59.886121 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1e4436585474a36b26f7ac87802a0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4436585474a36b26f7ac87802a0ce" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38115 } } }
I20260812 16:38:59.886107 7282 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e4436585474a36b26f7ac87802a0ce. Latest consensus state: current_term: 1 leader_uuid: "e1e4436585474a36b26f7ac87802a0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e4436585474a36b26f7ac87802a0ce" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 38115 } } }
I20260812 16:38:59.886538 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:59.886701 7282 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce [sys.catalog]: This master's current role is: LEADER
I20260812 16:38:59.888278 7287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:38:59.893061 7287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:38:59.903407 7287 catalog_manager.cc:1347] Generated new cluster ID: 60bbc1ab5bb243749c8498627037d674
I20260812 16:38:59.903544 7287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:38:59.942888 7287 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:38:59.944273 7287 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:38:59.955911 7287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce: Generated new TSK 0
I20260812 16:38:59.957118 7287 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab.
Entry for principal oryx/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab.
I20260812 16:39:00.031287 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:38115
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab
--principal=oryx/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:00.378165 7304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:00.378469 7304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:00.378608 7304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:00.386231 7304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:00.386358 7304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:00.386472 7304 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:39:00.386584 7304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:39:00.396834 7304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab
--principal=oryx/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.61.190:38115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:00.398177 7304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:00.399531 7304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:00.412604 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:00.412631 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:00.412575 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:00.413219 7304 server_base.cc:1034] running on GCE node
I20260812 16:39:00.413839 7304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:00.414747 7304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:00.415948 7304 hybrid_clock.cc:648] HybridClock initialized: now 1786552740415863 us; error 65 us; skew 500 ppm
I20260812 16:39:00.416232 7304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:00 dist-test-slave-f7th krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552740, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:00.419636 7304 init.cc:377] Logged in from keytab as oryx/127.31.61.129@KRBTEST.COM (short username oryx)
I20260812 16:39:00.421195 7304 webserver.cc:460] Webserver started at http://127.31.61.129:39431/ using document root <none> and password file <none>
I20260812 16:39:00.421890 7304 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:00.421993 7304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:00.422310 7304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:00.425397 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "4c6a692644a149999bbbd30974cba261"
format_stamp: "Formatted at 2026-08-12 16:39:00 on dist-test-slave-f7th"
server_key: "2a1f5dd2c05b206869018b27cd7aad09"
server_key_iv: "dbba04522117164d63ae17ecfcacb250"
server_key_version: "encryptionkey@0"
I20260812 16:39:00.426214 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "4c6a692644a149999bbbd30974cba261"
format_stamp: "Formatted at 2026-08-12 16:39:00 on dist-test-slave-f7th"
server_key: "2a1f5dd2c05b206869018b27cd7aad09"
server_key_iv: "dbba04522117164d63ae17ecfcacb250"
server_key_version: "encryptionkey@0"
I20260812 16:39:00.431545 7304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 16:39:00.435251 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:00.436990 7304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:39:00.437165 7304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "4c6a692644a149999bbbd30974cba261"
format_stamp: "Formatted at 2026-08-12 16:39:00 on dist-test-slave-f7th"
server_key: "2a1f5dd2c05b206869018b27cd7aad09"
server_key_iv: "dbba04522117164d63ae17ecfcacb250"
server_key_version: "encryptionkey@0"
I20260812 16:39:00.437387 7304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:00.450784 7304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:00.455147 7304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:00.455431 7304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:00.458662 7304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:00.458775 7304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:00.458969 7304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:00.459077 7304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:39:00 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552740, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:39:00.481020 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:00.459021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:36257 (local address 127.31.61.190:38115)
0812 16:39:00.459475 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:00.459482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:00.460742 (+ 1260us) server_negotiation.cc:407] Connection header received
0812 16:39:00.462845 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:00.462852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:00.462942 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:00.463042 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:00.465313 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:00.466309 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:00.469047 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:00.469450 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:00.469577 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:00.474614 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:00.474623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:00.474628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:00.474666 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:00.477544 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:00.478330 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:00.478337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:00.478342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:00.478517 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:00.478922 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:00.478927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:00.478931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:00.479322 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:00.479480 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:00.480321 (+ 841us) server_negotiation.cc:299] Negotiation successful
0812 16:39:00.480607 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":228,"threads_started":1}
I20260812 16:39:00.481841 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:00.459790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:38115 (local address 127.31.61.129:36257)
0812 16:39:00.460573 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:00.460620 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:00.462546 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:00.463344 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:00.463356 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:00.464189 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:00.465109 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:00.465143 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:00.466496 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:00.466502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:00.468810 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:00.468834 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:00.469718 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:00.469726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:00.469948 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:00.470700 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:00.470749 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:00.474291 (+ 3542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:00.477720 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:00.477727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:00.477741 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:00.478176 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:00.478639 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:00.478642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:00.478645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:00.478827 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:00.479434 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:00.479439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:00.479859 (+ 420us) client_negotiation.cc:769] Sending connection context
0812 16:39:00.480167 (+ 308us) client_negotiation.cc:240] Negotiation successful
0812 16:39:00.480652 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":304,"threads_started":1}
I20260812 16:39:00.502311 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:00.488544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:38115 (local address 127.31.61.129:42853)
0812 16:39:00.489280 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:00.489307 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:00.489477 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:00.490005 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:00.490008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:00.490695 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:00.491150 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:00.491162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:00.492448 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:00.492454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:00.493604 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:00.493618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:00.494410 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:00.494417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:00.494583 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:00.495218 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:00.495235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:00.496084 (+ 849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:00.499661 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:00.499667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:00.499671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:00.500238 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:00.500609 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:00.500614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:00.500618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:00.500756 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:00.501286 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:00.501291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:00.501516 (+ 225us) client_negotiation.cc:769] Sending connection context
0812 16:39:00.501699 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 16:39:00.501986 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":380,"threads_started":1}
I20260812 16:39:00.502475 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:00.488477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:42853 (local address 127.31.61.190:38115)
0812 16:39:00.488686 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:00.488693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:00.489408 (+ 715us) server_negotiation.cc:407] Connection header received
0812 16:39:00.489625 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:00.489631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:00.489719 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:00.489826 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:00.491309 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:00.492286 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:00.493760 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:00.494212 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:00.494328 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:00.496297 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:00.496302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:00.496305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:00.496337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:00.499463 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:00.500352 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:00.500358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:00.500363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:00.500498 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:00.500864 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:00.500870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:00.500873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:00.501175 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:00.501303 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:00.501850 (+ 547us) server_negotiation.cc:299] Negotiation successful
0812 16:39:00.502113 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 16:39:00.510350 7304 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:43999
I20260812 16:39:00.510368 7440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:43999 every 8 connection(s)
I20260812 16:39:00.511934 7304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:39:00.514103 7441 heartbeater.cc:347] Connected to a master server at 127.31.61.190:38115
I20260812 16:39:00.514329 7441 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:00.515013 7441 heartbeater.cc:511] Master 127.31.61.190:38115 requested a full tablet report, sending...
I20260812 16:39:00.517055 7231 ts_manager.cc:194] Registered new tserver with Master: 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999)
I20260812 16:39:00.518548 7231 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:36257
I20260812 16:39:00.518676 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 7304
I20260812 16:39:00.518882 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:39:00.519531 31990 external_mini_cluster.cc:1428] Setting key 003577f8ea710a42432ba10de7508723
WARNING: no policy specified for oryx/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab.
Entry for principal oryx/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab.
I20260812 16:39:00.573489 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:38115
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab
--principal=oryx/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:00.906486 7446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:00.906790 7446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:00.906895 7446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:00.914181 7446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:00.914296 7446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:00.914381 7446 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:39:00.914459 7446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:39:00.924381 7446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab
--principal=oryx/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.61.190:38115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:00.925616 7446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:00.926905 7446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:00.940176 7453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:00.940187 7452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:00.940284 7455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:00.941048 7446 server_base.cc:1034] running on GCE node
I20260812 16:39:00.941689 7446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:00.942625 7446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:00.943835 7446 hybrid_clock.cc:648] HybridClock initialized: now 1786552740943745 us; error 65 us; skew 500 ppm
I20260812 16:39:00.944128 7446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:00 dist-test-slave-f7th krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552740, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:00.947161 7446 init.cc:377] Logged in from keytab as oryx/127.31.61.130@KRBTEST.COM (short username oryx)
I20260812 16:39:00.948637 7446 webserver.cc:460] Webserver started at http://127.31.61.130:33749/ using document root <none> and password file <none>
I20260812 16:39:00.949314 7446 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:00.949415 7446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:00.949734 7446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:00.952651 7446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "4cc5b9bbf5704a09ad1f65c0265987ab"
format_stamp: "Formatted at 2026-08-12 16:39:00 on dist-test-slave-f7th"
server_key: "6618f9cede8f2e8781bbf6571765d4a1"
server_key_iv: "f12385556663e5219f9b958dbef8281e"
server_key_version: "encryptionkey@0"
I20260812 16:39:00.953447 7446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "4cc5b9bbf5704a09ad1f65c0265987ab"
format_stamp: "Formatted at 2026-08-12 16:39:00 on dist-test-slave-f7th"
server_key: "6618f9cede8f2e8781bbf6571765d4a1"
server_key_iv: "f12385556663e5219f9b958dbef8281e"
server_key_version: "encryptionkey@0"
I20260812 16:39:00.958604 7446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:39:00.962164 7462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:00.963495 7446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:39:00.963724 7446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "4cc5b9bbf5704a09ad1f65c0265987ab"
format_stamp: "Formatted at 2026-08-12 16:39:00 on dist-test-slave-f7th"
server_key: "6618f9cede8f2e8781bbf6571765d4a1"
server_key_iv: "f12385556663e5219f9b958dbef8281e"
server_key_version: "encryptionkey@0"
I20260812 16:39:00.963929 7446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:00.983568 7446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:00.988482 7446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:00.988759 7446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:00.991529 7446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:00.991639 7446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 16:39:00.991771 7446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:00.991830 7446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:39:01 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552740, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:39:01.016039 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:00.992128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:46625 (local address 127.31.61.190:38115)
0812 16:39:00.992333 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:00.992340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:00.993841 (+ 1501us) server_negotiation.cc:407] Connection header received
0812 16:39:00.996696 (+ 2855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:00.996702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:00.996808 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:00.996914 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:00.999718 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.000866 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.003403 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.004107 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.004252 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:01.009195 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:01.009204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:01.009209 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:01.009247 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:01.012289 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:01.013340 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:01.013347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:01.013351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:01.013506 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:01.014003 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:01.014009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:01.014013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:01.014335 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:01.014618 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:01.015380 (+ 762us) server_negotiation.cc:299] Negotiation successful
0812 16:39:01.015683 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 16:39:01.017561 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:00.992707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:38115 (local address 127.31.61.130:46625)
0812 16:39:00.993622 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:00.993679 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:00.996297 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:00.997295 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:00.997308 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:00.998296 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:00.999408 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:00.999441 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.001087 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.001106 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:01.003211 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:01.003226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.004365 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.004371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:01.004577 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:01.005374 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:01.005414 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:01.008929 (+ 3515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:01.012505 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:01.012521 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:01.012541 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:01.013189 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:01.013652 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:01.013657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:01.013660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:01.013879 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:01.014560 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:01.014579 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:01.014986 (+ 407us) client_negotiation.cc:769] Sending connection context
0812 16:39:01.015277 (+ 291us) client_negotiation.cc:240] Negotiation successful
0812 16:39:01.015894 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":377,"threads_started":1}
I20260812 16:39:01.040565 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:01.024079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:38115 (local address 127.31.61.130:37635)
0812 16:39:01.024750 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:01.024772 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:01.024952 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:01.025374 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:01.025379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:01.026294 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:01.026821 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:01.026838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.028078 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.028084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:01.029971 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:01.029990 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.031024 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.031032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:01.031220 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:01.032021 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:01.032052 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:01.033489 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:01.037301 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:01.037306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:01.037315 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:01.038000 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:01.038411 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:01.038417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:01.038421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:01.038640 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:01.039468 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:01.039473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:01.039730 (+ 257us) client_negotiation.cc:769] Sending connection context
0812 16:39:01.039912 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 16:39:01.040221 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":321,"threads_started":1}
I20260812 16:39:01.040622 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:01.023929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:37635 (local address 127.31.61.190:38115)
0812 16:39:01.024144 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:01.024150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:01.024895 (+ 745us) server_negotiation.cc:407] Connection header received
0812 16:39:01.025096 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:01.025103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:01.025212 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:01.025315 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:01.027002 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.027904 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.030142 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.030795 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.030939 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:01.033728 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:01.033733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:01.033737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:01.033771 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:01.037135 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:01.038134 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:01.038140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:01.038144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:01.038286 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:01.038781 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:01.038788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:01.038792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:01.039286 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:01.039486 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:01.040035 (+ 549us) server_negotiation.cc:299] Negotiation successful
0812 16:39:01.040295 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:39:01.042069 7446 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:41711
I20260812 16:39:01.042079 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:41711 every 8 connection(s)
I20260812 16:39:01.043702 7446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:39:01.045854 7585 heartbeater.cc:347] Connected to a master server at 127.31.61.190:38115
I20260812 16:39:01.046064 7585 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:01.046368 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 7446
I20260812 16:39:01.046515 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:39:01.046725 7585 heartbeater.cc:511] Master 127.31.61.190:38115 requested a full tablet report, sending...
I20260812 16:39:01.046993 31990 external_mini_cluster.cc:1428] Setting key 4c32d3e4f4a504adab91dc7d3d4ffe8b
I20260812 16:39:01.048111 7231 ts_manager.cc:194] Registered new tserver with Master: 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130:41711)
I20260812 16:39:01.048976 7231 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:46625
WARNING: no policy specified for oryx/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab.
Entry for principal oryx/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab.
I20260812 16:39:01.094861 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:38115
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab
--principal=oryx/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:01.432310 7589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:01.432602 7589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:01.432701 7589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:01.440141 7589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:01.440284 7589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:01.440410 7589 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 16:39:01.440516 7589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:39:01.450469 7589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab
--principal=oryx/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.61.190:38115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:01.451803 7589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:01.453131 7589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:01.466050 7594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:01.466074 7595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:01.465981 7597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:01.466869 7589 server_base.cc:1034] running on GCE node
I20260812 16:39:01.467541 7589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:01.468561 7589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:01.469775 7589 hybrid_clock.cc:648] HybridClock initialized: now 1786552741469699 us; error 55 us; skew 500 ppm
I20260812 16:39:01.470067 7589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:01 dist-test-slave-f7th krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552741, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:01.473141 7589 init.cc:377] Logged in from keytab as oryx/127.31.61.131@KRBTEST.COM (short username oryx)
I20260812 16:39:01.474531 7589 webserver.cc:460] Webserver started at http://127.31.61.131:35199/ using document root <none> and password file <none>
I20260812 16:39:01.475185 7589 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:01.475270 7589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:01.475546 7589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:01.478385 7589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "5b0c0624822744d2a899ce29094fc5b7"
format_stamp: "Formatted at 2026-08-12 16:39:01 on dist-test-slave-f7th"
server_key: "24b48ea347636bf7485c6043f709e987"
server_key_iv: "5280411ce44cc76bb37345e0727bfda9"
server_key_version: "encryptionkey@0"
I20260812 16:39:01.479163 7589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "5b0c0624822744d2a899ce29094fc5b7"
format_stamp: "Formatted at 2026-08-12 16:39:01 on dist-test-slave-f7th"
server_key: "24b48ea347636bf7485c6043f709e987"
server_key_iv: "5280411ce44cc76bb37345e0727bfda9"
server_key_version: "encryptionkey@0"
I20260812 16:39:01.484529 7589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 16:39:01.488179 7605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:01.489599 7589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:39:01.489785 7589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "5b0c0624822744d2a899ce29094fc5b7"
format_stamp: "Formatted at 2026-08-12 16:39:01 on dist-test-slave-f7th"
server_key: "24b48ea347636bf7485c6043f709e987"
server_key_iv: "5280411ce44cc76bb37345e0727bfda9"
server_key_version: "encryptionkey@0"
I20260812 16:39:01.490028 7589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:01.505720 7589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:01.510044 7589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:01.510308 7589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:01.513078 7589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:01.513182 7589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:01.513345 7589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:01.513410 7589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:01.521708 7441 heartbeater.cc:503] Master 127.31.61.190:38115 was elected leader, sending a full tablet report...
Aug 12 16:39:01 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552741, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:39:01.536670 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:01.513875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:54063 (local address 127.31.61.190:38115)
0812 16:39:01.514060 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:01.514066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:01.515639 (+ 1573us) server_negotiation.cc:407] Connection header received
0812 16:39:01.518498 (+ 2859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:01.518503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:01.518600 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:01.518689 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:01.521147 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.522125 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.524820 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.525236 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.525350 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:01.529922 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:01.529928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:01.529932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:01.529967 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:01.532974 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:01.534051 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:01.534067 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:01.534071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:01.534228 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:01.534741 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:01.534747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:01.534750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:01.535046 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:01.535189 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:01.536044 (+ 855us) server_negotiation.cc:299] Negotiation successful
0812 16:39:01.536303 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:39:01.538250 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:01.514456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:38115 (local address 127.31.61.131:54063)
0812 16:39:01.515416 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:01.515466 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:01.518147 (+ 2681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:01.518983 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:01.518995 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:01.519855 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:01.520904 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:01.520933 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.522342 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.522347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:01.524633 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:01.524655 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.525490 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.525503 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:01.525844 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:01.526649 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:01.526673 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:01.529643 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:01.533130 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:01.533140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:01.533155 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:01.533865 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:01.534346 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:01.534352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:01.534356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:01.534614 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:01.535192 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:01.535201 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:01.535618 (+ 417us) client_negotiation.cc:769] Sending connection context
0812 16:39:01.535902 (+ 284us) client_negotiation.cc:240] Negotiation successful
0812 16:39:01.536502 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":455,"threads_started":1}
I20260812 16:39:01.554127 7589 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:41973
I20260812 16:39:01.554148 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:41973 every 8 connection(s)
I20260812 16:39:01.555537 7589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:39:01.558156 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 7589
I20260812 16:39:01.558306 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:39:01.558887 31990 external_mini_cluster.cc:1428] Setting key 0e9ea4896d4941dd62764a69dd23c3ad
I20260812 16:39:01.558943 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:01.545305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:51947 (local address 127.31.61.190:38115)
0812 16:39:01.545498 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:01.545504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:01.546096 (+ 592us) server_negotiation.cc:407] Connection header received
0812 16:39:01.546278 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:01.546282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:01.546345 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:01.546424 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:01.547961 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.548931 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.550868 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.551299 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.551416 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:01.553603 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:01.553609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:01.553614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:01.553648 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:01.556251 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:01.556904 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:01.556909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:01.556913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:01.557043 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:01.557489 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:01.557494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:01.557497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:01.557847 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:01.557991 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:01.558387 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 16:39:01.558616 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:39:01.557749 7728 heartbeater.cc:347] Connected to a master server at 127.31.61.190:38115
I20260812 16:39:01.558840 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:01.545265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:38115 (local address 127.31.61.131:51947)
0812 16:39:01.545958 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:01.545983 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:01.546160 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:01.546484 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:01.546489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:01.547239 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:01.547765 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:01.547780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.549135 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.549138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:01.550721 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:01.550737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.551507 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.551513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:01.551731 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:01.552413 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:01.552431 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:01.553357 (+ 926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:01.556375 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:01.556381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:01.556385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:01.556793 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:01.557185 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:01.557189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:01.557193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:01.557361 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:01.557973 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:01.557977 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:01.558146 (+ 169us) client_negotiation.cc:769] Sending connection context
0812 16:39:01.558273 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 16:39:01.558559 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":387,"threads_started":1}
I20260812 16:39:01.559406 7728 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:01.560228 7728 heartbeater.cc:511] Master 127.31.61.190:38115 requested a full tablet report, sending...
I20260812 16:39:01.561863 7230 ts_manager.cc:194] Registered new tserver with Master: 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973)
I20260812 16:39:01.562987 7230 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:54063
I20260812 16:39:01.574760 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 16:39:01 dist-test-slave-f7th krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 16:39:01 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:01 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:02.051821 7585 heartbeater.cc:503] Master 127.31.61.190:38115 was elected leader, sending a full tablet report...
I20260812 16:39:02.565483 7728 heartbeater.cc:503] Master 127.31.61.190:38115 was elected leader, sending a full tablet report...
W20260812 16:39:02.654114 7737 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.034s user 0.009s sys 0.005s
W20260812 16:39:02.655314 7327 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:01.614135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.31.61.190:38115)
0812 16:39:01.614346 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:01.614353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:01.614583 (+ 230us) server_negotiation.cc:407] Connection header received
0812 16:39:01.614762 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:01.614766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:01.614832 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:01.614919 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:01.616308 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.617030 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.618237 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:01.618685 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:01.618810 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.655065 (+1036255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54344: BlockingRecv error: recv got EOF from 127.0.0.1:54344 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 12 16:39:02 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:39:02.679636 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.661430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.31.61.190:38115)
0812 16:39:02.661667 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:02.661674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:02.661792 (+ 118us) server_negotiation.cc:407] Connection header received
0812 16:39:02.662020 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:02.662025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:02.662105 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:02.662194 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:02.663351 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.664186 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.665412 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.665814 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.665929 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.673974 (+ 8045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:02.673981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:02.673984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:02.674027 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:02.676934 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:02.677665 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:02.677671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:02.677673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:02.677808 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:02.678183 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:02.678187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:02.678189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:02.678484 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:02.678749 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:02.679050 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 16:39:02.679310 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 16:39:02.684497 7230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:39:02.691800 7230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:39:02.717620 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.705488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.31.61.131:41973)
0812 16:39:02.706280 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:02.706289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:02.707019 (+ 730us) server_negotiation.cc:407] Connection header received
0812 16:39:02.707753 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:02.707762 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:02.707996 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:02.708155 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:02.709242 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.710917 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.714134 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.715807 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.715935 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.716196 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:02.717098 (+ 902us) server_negotiation.cc:299] Negotiation successful
0812 16:39:02.717297 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":469,"threads_started":1}
I20260812 16:39:02.718539 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.705261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:41973 (local address 127.0.0.1:40914)
0812 16:39:02.706875 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:02.706919 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:02.707518 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:02.708453 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:02.708461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:02.708497 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:02.708996 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.709010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.711145 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.711152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.713919 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.713938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.716549 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.716556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.716757 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.716871 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 16:39:02.717818 (+ 947us) client_negotiation.cc:240] Negotiation successful
0812 16:39:02.718131 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1281,"mutex_wait_us":32,"thread_start_us":1089,"threads_started":1}
I20260812 16:39:02.718708 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.706515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.31.61.129:43999)
0812 16:39:02.707139 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:02.707146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:02.707172 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:39:02.707557 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:02.707566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:02.707847 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:02.708446 (+ 599us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:02.709472 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.711518 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.715992 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.717443 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.717568 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.717763 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:02.718229 (+ 466us) server_negotiation.cc:299] Negotiation successful
0812 16:39:02.718398 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":418,"threads_started":1}
I20260812 16:39:02.718623 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.705890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:43999 (local address 127.0.0.1:51786)
0812 16:39:02.707096 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:02.707112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:02.707272 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:02.708655 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:02.708660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:02.708687 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:02.709244 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.709260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.711781 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.711787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.715795 (+ 4008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.715811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.717642 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.717648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.718013 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.718044 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 16:39:02.718146 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 16:39:02.718271 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":1695,"thread_start_us":789,"threads_started":1}
I20260812 16:39:02.722754 7371 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7ae94cb1904adea5556b7d914b50fc (DEFAULT_TABLE table=test-table [id=fb5a712405c6400d8fec035b92da254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:02.722749 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7ae94cb1904adea5556b7d914b50fc (DEFAULT_TABLE table=test-table [id=fb5a712405c6400d8fec035b92da254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:02.723450 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.707191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:41711 (local address 127.0.0.1:33492)
0812 16:39:02.715541 (+ 8350us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:02.715562 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:02.715722 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:02.716703 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:02.716708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:02.716736 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:02.717154 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.717167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.718852 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.718857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.720786 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.720803 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.722623 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.722630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.722802 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.722837 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:39:02.722937 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 16:39:02.723079 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8192,"mutex_wait_us":2,"thread_start_us":7991,"threads_started":1}
I20260812 16:39:02.723456 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.707765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.31.61.130:41711)
0812 16:39:02.715791 (+ 8026us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:02.715800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:02.715831 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:39:02.715937 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:02.715945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:02.716246 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:02.716480 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:02.717328 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.718686 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.720953 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.722393 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.722551 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.722756 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:02.722997 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 16:39:02.723174 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7842,"thread_start_us":8044,"threads_started":1}
I20260812 16:39:02.724377 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7ae94cb1904adea5556b7d914b50fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:02.724539 7371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7ae94cb1904adea5556b7d914b50fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:02.726653 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7ae94cb1904adea5556b7d914b50fc (DEFAULT_TABLE table=test-table [id=fb5a712405c6400d8fec035b92da254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:02.727963 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7ae94cb1904adea5556b7d914b50fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:02.740357 7755 tablet_bootstrap.cc:492] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: Bootstrap starting.
I20260812 16:39:02.742532 7757 tablet_bootstrap.cc:492] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: Bootstrap starting.
I20260812 16:39:02.744576 7755 tablet_bootstrap.cc:654] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:02.746285 7757 tablet_bootstrap.cc:654] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:02.746373 7755 log.cc:824] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:02.747771 7757 log.cc:824] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:02.748003 7756 tablet_bootstrap.cc:492] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: Bootstrap starting.
I20260812 16:39:02.750092 7755 tablet_bootstrap.cc:492] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: No bootstrap required, opened a new log
I20260812 16:39:02.750351 7755 ts_tablet_manager.cc:1397] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 16:39:02.751886 7756 tablet_bootstrap.cc:654] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:02.753072 7756 log.cc:824] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:02.754523 7757 tablet_bootstrap.cc:492] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: No bootstrap required, opened a new log
I20260812 16:39:02.754842 7757 ts_tablet_manager.cc:1397] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260812 16:39:02.758579 7755 raft_consensus.cc:348] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:02.758941 7755 raft_consensus.cc:374] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:02.758982 7756 tablet_bootstrap.cc:492] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: No bootstrap required, opened a new log
I20260812 16:39:02.759064 7755 raft_consensus.cc:729] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6a692644a149999bbbd30974cba261, State: Initialized, Role: FOLLOWER
I20260812 16:39:02.759284 7756 ts_tablet_manager.cc:1397] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260812 16:39:02.759693 7755 consensus_queue.cc:260] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:02.761219 7755 ts_tablet_manager.cc:1428] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260812 16:39:02.764276 7757 raft_consensus.cc:348] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:02.764647 7757 raft_consensus.cc:374] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:02.764750 7757 raft_consensus.cc:729] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc5b9bbf5704a09ad1f65c0265987ab, State: Initialized, Role: FOLLOWER
W20260812 16:39:02.765312 7442 tablet.cc:2369] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:39:02.766201 7757 consensus_queue.cc:260] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:02.769002 7757 ts_tablet_manager.cc:1428] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260812 16:39:02.769569 7756 raft_consensus.cc:348] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:02.770072 7756 raft_consensus.cc:374] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:02.770192 7756 raft_consensus.cc:729] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0c0624822744d2a899ce29094fc5b7, State: Initialized, Role: FOLLOWER
I20260812 16:39:02.770819 7756 consensus_queue.cc:260] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:02.772406 7756 ts_tablet_manager.cc:1428] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260812 16:39:02.797008 7586 tablet.cc:2369] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:39:02.808732 7729 tablet.cc:2369] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:39:02.910454 7763 raft_consensus.cc:482] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:39:02.910779 7763 raft_consensus.cc:504] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:02.912498 7763 leader_election.cc:288] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973), 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999)
I20260812 16:39:02.919158 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.912895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:47843 (local address 127.31.61.131:41973)
0812 16:39:02.913167 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:02.913175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:02.913955 (+ 780us) server_negotiation.cc:407] Connection header received
0812 16:39:02.914133 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:02.914139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:02.914211 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:02.914277 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:02.914890 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.915671 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.917515 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.918272 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.918431 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.918505 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:02.918804 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 16:39:02.918932 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 16:39:02.919176 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.912914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:47471 (local address 127.31.61.129:43999)
0812 16:39:02.913190 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:02.913198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:02.914020 (+ 822us) server_negotiation.cc:407] Connection header received
0812 16:39:02.914178 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:02.914181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:02.914241 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:02.914327 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:02.914967 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.915718 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.917487 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.918273 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.918423 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.918500 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:02.918815 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 16:39:02.918940 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 16:39:02.919435 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.912924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:41973 (local address 127.31.61.130:47843)
0812 16:39:02.913839 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:02.913854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:02.913987 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:02.914318 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:02.914322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:02.914346 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:02.914749 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.914759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.915788 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.915794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.917355 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.917367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.918507 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.918518 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.918649 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.918674 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 16:39:02.918749 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 16:39:02.918876 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":5,"thread_start_us":542,"threads_started":1}
I20260812 16:39:02.919579 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.912940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:43999 (local address 127.31.61.130:47471)
0812 16:39:02.913891 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:02.913914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:02.914051 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:02.914359 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:02.914363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:02.914386 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:02.914803 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.914815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.915859 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.915862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.917329 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:02.917344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.918541 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.918547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:02.918668 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.918690 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 16:39:02.918775 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 16:39:02.918899 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":22,"thread_start_us":570,"threads_started":1}
I20260812 16:39:02.920600 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7ae94cb1904adea5556b7d914b50fc" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6a692644a149999bbbd30974cba261" is_pre_election: true
I20260812 16:39:02.920581 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7ae94cb1904adea5556b7d914b50fc" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7" is_pre_election: true
I20260812 16:39:02.921183 7676 raft_consensus.cc:2454] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 0.
I20260812 16:39:02.921182 7396 raft_consensus.cc:2454] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 0.
I20260812 16:39:02.921866 7463 leader_election.cc:302] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc5b9bbf5704a09ad1f65c0265987ab, 5b0c0624822744d2a899ce29094fc5b7; no voters:
I20260812 16:39:02.922318 7763 raft_consensus.cc:2781] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:39:02.922468 7763 raft_consensus.cc:482] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:39:02.922591 7763 raft_consensus.cc:3037] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:02.924817 7763 raft_consensus.cc:504] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:02.925379 7763 leader_election.cc:288] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 election: Requested vote from peers 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973), 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999)
I20260812 16:39:02.925971 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7ae94cb1904adea5556b7d914b50fc" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7"
I20260812 16:39:02.925971 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7ae94cb1904adea5556b7d914b50fc" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6a692644a149999bbbd30974cba261"
I20260812 16:39:02.926199 7676 raft_consensus.cc:3037] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:02.926199 7396 raft_consensus.cc:3037] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:02.929358 7396 raft_consensus.cc:2454] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 1.
I20260812 16:39:02.929358 7676 raft_consensus.cc:2454] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 1.
I20260812 16:39:02.929860 7463 leader_election.cc:302] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c6a692644a149999bbbd30974cba261, 4cc5b9bbf5704a09ad1f65c0265987ab, 5b0c0624822744d2a899ce29094fc5b7; no voters:
I20260812 16:39:02.930179 7763 raft_consensus.cc:2781] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:02.930615 7763 raft_consensus.cc:686] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 LEADER]: Becoming Leader. State: Replica: 4cc5b9bbf5704a09ad1f65c0265987ab, State: Running, Role: LEADER
I20260812 16:39:02.931078 7763 consensus_queue.cc:237] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:02.936026 7230 catalog_manager.cc:5495] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130). New cstate: current_term: 1 leader_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20260812 16:39:02.954524 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:02.949289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.31.61.130:41711)
0812 16:39:02.949497 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:02.949504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:02.949581 (+ 77us) server_negotiation.cc:407] Connection header received
0812 16:39:02.949834 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:02.949838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:02.949918 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:02.950047 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:39:02.950423 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.951126 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.952463 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:02.952873 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:02.952982 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:02.953419 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:39:02.953944 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:39:02.954090 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:02.954179 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 16:39:02.954266 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 16:39:02.962522 7396 raft_consensus.cc:1261] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Refusing update from remote peer 4cc5b9bbf5704a09ad1f65c0265987ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:02.962520 7676 raft_consensus.cc:1261] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Refusing update from remote peer 4cc5b9bbf5704a09ad1f65c0265987ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:02.963510 7767 consensus_queue.cc:1035] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:02.963894 7763 consensus_queue.cc:1035] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:02.990437 7230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:02.990723 7230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:02.994443 7230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce: Sending DeleteTablet for 3 replicas of tablet 7b7ae94cb1904adea5556b7d914b50fc
I20260812 16:39:02.995676 7371 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7ae94cb1904adea5556b7d914b50fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:02 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:51786
I20260812 16:39:02.995687 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7ae94cb1904adea5556b7d914b50fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:02 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:40914
I20260812 16:39:02.996747 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7ae94cb1904adea5556b7d914b50fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:02 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:33492
I20260812 16:39:02.996712 7230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:39:02.997713 7230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:39:03.007015 7654 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe339daeb8c4bae9e9ab9f2a98be24a (DEFAULT_TABLE table=test-table [id=7fd0aaf6766748f4aab5790b6af1fa50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:03.007124 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe339daeb8c4bae9e9ab9f2a98be24a (DEFAULT_TABLE table=test-table [id=7fd0aaf6766748f4aab5790b6af1fa50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:03.007673 7654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe339daeb8c4bae9e9ab9f2a98be24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.007804 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe339daeb8c4bae9e9ab9f2a98be24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.013033 7370 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe339daeb8c4bae9e9ab9f2a98be24a (DEFAULT_TABLE table=test-table [id=7fd0aaf6766748f4aab5790b6af1fa50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:03.013705 7370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe339daeb8c4bae9e9ab9f2a98be24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.014398 7756 tablet_bootstrap.cc:492] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: Bootstrap starting.
I20260812 16:39:03.016166 7756 tablet_bootstrap.cc:654] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.016429 7782 tablet_replica.cc:330] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: stopping tablet replica
I20260812 16:39:03.016912 7782 raft_consensus.cc:2229] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:39:03.017659 7782 raft_consensus.cc:2258] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.019922 7782 ts_tablet_manager.cc:1905] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.023187 7755 tablet_bootstrap.cc:492] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: Bootstrap starting.
I20260812 16:39:03.024904 7755 tablet_bootstrap.cc:654] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.030763 7757 tablet_bootstrap.cc:492] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: Bootstrap starting.
I20260812 16:39:03.032555 7757 tablet_bootstrap.cc:654] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.034202 7782 ts_tablet_manager.cc:1918] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:39:03.034391 7782 log.cc:1196] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/7b7ae94cb1904adea5556b7d914b50fc
I20260812 16:39:03.034749 7757 tablet_bootstrap.cc:492] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: No bootstrap required, opened a new log
I20260812 16:39:03.034906 7757 ts_tablet_manager.cc:1397] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260812 16:39:03.034936 7782 ts_tablet_manager.cc:1939] T 7b7ae94cb1904adea5556b7d914b50fc P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting consensus metadata
I20260812 16:39:03.035868 7757 raft_consensus.cc:348] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.036105 7757 raft_consensus.cc:374] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.036194 7757 raft_consensus.cc:729] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc5b9bbf5704a09ad1f65c0265987ab, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.037926 7757 consensus_queue.cc:260] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.042781 7757 ts_tablet_manager.cc:1428] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260812 16:39:03.043191 7756 tablet_bootstrap.cc:492] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: No bootstrap required, opened a new log
I20260812 16:39:03.043356 7756 ts_tablet_manager.cc:1397] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.004s
I20260812 16:39:03.044193 7756 raft_consensus.cc:348] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.044430 7756 raft_consensus.cc:374] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.044514 7756 raft_consensus.cc:729] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0c0624822744d2a899ce29094fc5b7, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.045496 7755 tablet_bootstrap.cc:492] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: No bootstrap required, opened a new log
I20260812 16:39:03.045650 7755 ts_tablet_manager.cc:1397] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.009s
I20260812 16:39:03.046451 7755 raft_consensus.cc:348] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.046682 7755 raft_consensus.cc:374] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.046774 7755 raft_consensus.cc:729] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6a692644a149999bbbd30974cba261, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.047024 7755 consensus_queue.cc:260] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.047766 7781 tablet_replica.cc:330] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: stopping tablet replica
I20260812 16:39:03.048285 7781 raft_consensus.cc:2229] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:03.048779 7781 raft_consensus.cc:2258] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.048915 7215 catalog_manager.cc:4841] TS 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130:41711): tablet 7b7ae94cb1904adea5556b7d914b50fc (table test-table [id=fb5a712405c6400d8fec035b92da254d]) successfully deleted
I20260812 16:39:03.049598 7755 ts_tablet_manager.cc:1428] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260812 16:39:03.050268 7756 consensus_queue.cc:260] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.052289 7781 ts_tablet_manager.cc:1905] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.054415 7767 raft_consensus.cc:482] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:39:03.054636 7767 raft_consensus.cc:504] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.055307 7767 leader_election.cc:288] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999), 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973)
I20260812 16:39:03.056430 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe339daeb8c4bae9e9ab9f2a98be24a" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6a692644a149999bbbd30974cba261" is_pre_election: true
I20260812 16:39:03.056701 7396 raft_consensus.cc:2454] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 0.
I20260812 16:39:03.057229 7464 leader_election.cc:302] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6a692644a149999bbbd30974cba261, 4cc5b9bbf5704a09ad1f65c0265987ab; no voters:
I20260812 16:39:03.057994 7767 raft_consensus.cc:2781] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:39:03.058100 7767 raft_consensus.cc:482] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:39:03.058017 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe339daeb8c4bae9e9ab9f2a98be24a" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7" is_pre_election: true
I20260812 16:39:03.058200 7767 raft_consensus.cc:3037] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.058274 7676 raft_consensus.cc:2454] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 0.
I20260812 16:39:03.059161 7756 ts_tablet_manager.cc:1428] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20260812 16:39:03.060740 7780 tablet_replica.cc:330] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: stopping tablet replica
I20260812 16:39:03.061256 7780 raft_consensus.cc:2229] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:03.061770 7780 raft_consensus.cc:2258] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.063863 7767 raft_consensus.cc:504] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.064795 7767 leader_election.cc:288] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 election: Requested vote from peers 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999), 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973)
I20260812 16:39:03.065021 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe339daeb8c4bae9e9ab9f2a98be24a" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6a692644a149999bbbd30974cba261"
I20260812 16:39:03.065243 7396 raft_consensus.cc:3037] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.065302 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe339daeb8c4bae9e9ab9f2a98be24a" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7"
I20260812 16:39:03.065497 7676 raft_consensus.cc:3037] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.066686 7781 ts_tablet_manager.cc:1918] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:39:03.066840 7781 log.cc:1196] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/7b7ae94cb1904adea5556b7d914b50fc
I20260812 16:39:03.067445 7781 ts_tablet_manager.cc:1939] T 7b7ae94cb1904adea5556b7d914b50fc P 5b0c0624822744d2a899ce29094fc5b7: Deleting consensus metadata
I20260812 16:39:03.068001 7396 raft_consensus.cc:2454] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 1.
I20260812 16:39:03.068563 7676 raft_consensus.cc:2454] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 1.
I20260812 16:39:03.068871 7464 leader_election.cc:302] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6a692644a149999bbbd30974cba261, 4cc5b9bbf5704a09ad1f65c0265987ab; no voters:
I20260812 16:39:03.069212 7767 raft_consensus.cc:2781] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:03.069380 7767 raft_consensus.cc:686] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 LEADER]: Becoming Leader. State: Replica: 4cc5b9bbf5704a09ad1f65c0265987ab, State: Running, Role: LEADER
I20260812 16:39:03.069943 7780 ts_tablet_manager.cc:1905] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.069850 7767 consensus_queue.cc:237] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.072562 7215 catalog_manager.cc:4841] TS 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973): tablet 7b7ae94cb1904adea5556b7d914b50fc (table test-table [id=fb5a712405c6400d8fec035b92da254d]) successfully deleted
I20260812 16:39:03.074776 7230 catalog_manager.cc:5495] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130). New cstate: current_term: 1 leader_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } health_report { overall_health: UNKNOWN } } }
I20260812 16:39:03.076835 7780 ts_tablet_manager.cc:1918] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:39:03.076980 7780 log.cc:1196] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/7b7ae94cb1904adea5556b7d914b50fc
I20260812 16:39:03.077520 7780 ts_tablet_manager.cc:1939] T 7b7ae94cb1904adea5556b7d914b50fc P 4c6a692644a149999bbbd30974cba261: Deleting consensus metadata
I20260812 16:39:03.079279 7216 catalog_manager.cc:4841] TS 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999): tablet 7b7ae94cb1904adea5556b7d914b50fc (table test-table [id=fb5a712405c6400d8fec035b92da254d]) successfully deleted
Aug 12 16:39:03 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552739, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.190@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:39:03.129199 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.114292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.31.61.190:38115)
0812 16:39:03.114506 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.114514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.115071 (+ 557us) server_negotiation.cc:407] Connection header received
0812 16:39:03.115247 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.115250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.115307 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.115385 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:03.116637 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.117415 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.118641 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.119038 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.119165 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.123228 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:03.123235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:03.123239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:03.123271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:03.126338 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:03.127108 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:03.127112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:03.127114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:03.127232 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:03.127643 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:03.127648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:03.127650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:03.127945 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:03.128111 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.128539 (+ 428us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.128849 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":73}
I20260812 16:39:03.129228 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.114286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:38115 (local address 127.0.0.1:40760)
0812 16:39:03.114958 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:03.114982 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:03.115133 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:03.115438 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:03.115442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:03.116195 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:03.116490 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.116502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.117541 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.117547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.118502 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.118517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.119232 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.119239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.119402 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.120228 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:03.120247 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:03.123016 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:03.126508 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:03.126546 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:03.126552 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:03.126978 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:03.127341 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:03.127344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:03.127346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:03.127485 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:03.128089 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:03.128097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:03.128261 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 16:39:03.128413 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 16:39:03.128713 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":397,"threads_started":1}
I20260812 16:39:03.135087 7230 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:40760:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 16:39:03.136325 7230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:39:03.144972 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0a958d1dc349248ca0b2c51273678f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a844c09918834702b888185e8e8da94b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:39:03.144944 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0a958d1dc349248ca0b2c51273678f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a844c09918834702b888185e8e8da94b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:39:03.145272 7371 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0a958d1dc349248ca0b2c51273678f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a844c09918834702b888185e8e8da94b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 16:39:03.145570 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0a958d1dc349248ca0b2c51273678f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.145576 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0a958d1dc349248ca0b2c51273678f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.145787 7371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0a958d1dc349248ca0b2c51273678f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.152907 7755 tablet_bootstrap.cc:492] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: Bootstrap starting.
I20260812 16:39:03.153096 7757 tablet_bootstrap.cc:492] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: Bootstrap starting.
I20260812 16:39:03.154476 7755 tablet_bootstrap.cc:654] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.154922 7757 tablet_bootstrap.cc:654] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.156818 7755 tablet_bootstrap.cc:492] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: No bootstrap required, opened a new log
I20260812 16:39:03.156988 7755 ts_tablet_manager.cc:1397] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260812 16:39:03.157747 7755 raft_consensus.cc:348] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:03.157997 7755 raft_consensus.cc:374] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.158082 7755 raft_consensus.cc:729] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6a692644a149999bbbd30974cba261, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.158377 7755 consensus_queue.cc:260] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:03.159029 7755 ts_tablet_manager.cc:1428] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 16:39:03.158878 7762 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.159273 7762 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:39:03.161649 7756 tablet_bootstrap.cc:492] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: Bootstrap starting.
I20260812 16:39:03.163867 7756 tablet_bootstrap.cc:654] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.166944 7756 tablet_bootstrap.cc:492] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: No bootstrap required, opened a new log
I20260812 16:39:03.166944 7757 tablet_bootstrap.cc:492] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: No bootstrap required, opened a new log
I20260812 16:39:03.167119 7756 ts_tablet_manager.cc:1397] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260812 16:39:03.167135 7757 ts_tablet_manager.cc:1397] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20260812 16:39:03.167867 7756 raft_consensus.cc:348] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:03.168120 7756 raft_consensus.cc:374] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.168201 7756 raft_consensus.cc:729] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0c0624822744d2a899ce29094fc5b7, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.168076 7757 raft_consensus.cc:348] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:03.168427 7757 raft_consensus.cc:374] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.168535 7757 raft_consensus.cc:729] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc5b9bbf5704a09ad1f65c0265987ab, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.168443 7756 consensus_queue.cc:260] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:03.168804 7757 consensus_queue.cc:260] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:03.169031 7756 ts_tablet_manager.cc:1428] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 16:39:03.169242 7767 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.169526 7767 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:39:03.169356 7764 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.169703 7764 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:39:03.172937 7757 ts_tablet_manager.cc:1428] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260812 16:39:03.251044 7762 raft_consensus.cc:482] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:39:03.251271 7762 raft_consensus.cc:504] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:03.252807 7762 leader_election.cc:288] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130:41711), 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973)
I20260812 16:39:03.260700 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.253172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:41711 (local address 127.31.61.129:35029)
0812 16:39:03.253720 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:03.253739 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:03.253881 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:03.254528 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:03.254534 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:03.254567 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:03.254979 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.254991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.256636 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.256641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.258542 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.258554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.259900 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.259907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.260034 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.260061 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 16:39:03.260158 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 16:39:03.260297 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":26,"thread_start_us":276,"threads_started":1}
I20260812 16:39:03.261016 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.253310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:35029 (local address 127.31.61.130:41711)
0812 16:39:03.253517 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.253524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.253827 (+ 303us) server_negotiation.cc:407] Connection header received
0812 16:39:03.254106 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.254113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.254224 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.254297 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:03.255201 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.256465 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.258718 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.259669 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.259819 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.259905 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.260214 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.260366 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:39:03.262627 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.254199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:58219 (local address 127.31.61.131:41973)
0812 16:39:03.254379 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.254386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.255697 (+ 1311us) server_negotiation.cc:407] Connection header received
0812 16:39:03.255851 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.255855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.255914 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.255985 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:03.256560 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.257675 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.260156 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.261430 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.261598 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.261670 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.262214 (+ 544us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.262355 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:39:03.262318 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0a958d1dc349248ca0b2c51273678f" candidate_uuid: "4c6a692644a149999bbbd30974cba261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" is_pre_election: true
I20260812 16:39:03.262892 7539 raft_consensus.cc:2454] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6a692644a149999bbbd30974cba261 in term 0.
I20260812 16:39:03.263657 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.253892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:41973 (local address 127.31.61.129:58219)
0812 16:39:03.255576 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:03.255591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:03.255731 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:03.256016 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:03.256021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:03.256045 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:03.256373 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.256382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.257828 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.257834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.259981 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.259997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.261796 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.261803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.261991 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.262025 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:39:03.263068 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0812 16:39:03.263235 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1588,"mutex_wait_us":4,"thread_start_us":1388,"threads_started":1}
I20260812 16:39:03.263535 7320 leader_election.cc:302] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6a692644a149999bbbd30974cba261, 4cc5b9bbf5704a09ad1f65c0265987ab; no voters:
I20260812 16:39:03.263994 7762 raft_consensus.cc:2781] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:39:03.264112 7762 raft_consensus.cc:482] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:39:03.264199 7762 raft_consensus.cc:3037] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.266311 7762 raft_consensus.cc:504] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:03.267023 7762 leader_election.cc:288] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130:41711), 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973)
I20260812 16:39:03.267527 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0a958d1dc349248ca0b2c51273678f" candidate_uuid: "4c6a692644a149999bbbd30974cba261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab"
I20260812 16:39:03.267755 7539 raft_consensus.cc:3037] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.267994 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0a958d1dc349248ca0b2c51273678f" candidate_uuid: "4c6a692644a149999bbbd30974cba261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7" is_pre_election: true
I20260812 16:39:03.268277 7676 raft_consensus.cc:2454] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6a692644a149999bbbd30974cba261 in term 0.
I20260812 16:39:03.269097 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0a958d1dc349248ca0b2c51273678f" candidate_uuid: "4c6a692644a149999bbbd30974cba261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7"
I20260812 16:39:03.269310 7676 raft_consensus.cc:3037] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.270360 7539 raft_consensus.cc:2454] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6a692644a149999bbbd30974cba261 in term 1.
I20260812 16:39:03.271801 7320 leader_election.cc:302] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6a692644a149999bbbd30974cba261, 4cc5b9bbf5704a09ad1f65c0265987ab; no voters:
I20260812 16:39:03.272087 7762 raft_consensus.cc:2781] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:03.272254 7676 raft_consensus.cc:2454] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6a692644a149999bbbd30974cba261 in term 1.
I20260812 16:39:03.272701 7762 raft_consensus.cc:686] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [term 1 LEADER]: Becoming Leader. State: Replica: 4c6a692644a149999bbbd30974cba261, State: Running, Role: LEADER
I20260812 16:39:03.273128 7762 consensus_queue.cc:237] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } }
I20260812 16:39:03.275898 7798 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: TxnStatusTablet state changed. Reason: New leader 4c6a692644a149999bbbd30974cba261. Latest consensus state: current_term: 1 leader_uuid: "4c6a692644a149999bbbd30974cba261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.276257 7798 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:39:03.276669 7230 catalog_manager.cc:5495] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6a692644a149999bbbd30974cba261 (127.31.61.129). New cstate: current_term: 1 leader_uuid: "4c6a692644a149999bbbd30974cba261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } health_report { overall_health: HEALTHY } } }
I20260812 16:39:03.276964 7800 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 16:39:03.277184 7800 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 16:39:03.302380 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.295354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.31.61.129:43999)
0812 16:39:03.295547 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.295553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.295574 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:39:03.295698 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.295704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.295817 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.295976 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:39:03.296392 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.297640 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.299582 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.300286 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.300433 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.300963 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:39:03.301653 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:39:03.301806 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.301939 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.302029 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:39:03.302383 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.295028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:43999 (local address 127.0.0.1:51800)
0812 16:39:03.295282 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:03.295303 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:03.295443 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:03.296006 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:03.296011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:03.296025 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:39:03.296232 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.296245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.297779 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.297786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.299421 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.299438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.300462 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.300469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.300676 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.300755 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:39:03.301778 (+ 1023us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:39:03.301810 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 16:39:03.301915 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 16:39:03.302079 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 16:39:03.307406 7676 raft_consensus.cc:1261] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Refusing update from remote peer 4c6a692644a149999bbbd30974cba261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:03.307693 7539 raft_consensus.cc:1261] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Refusing update from remote peer 4c6a692644a149999bbbd30974cba261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:03.308626 7762 consensus_queue.cc:1035] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:03.309337 7798 consensus_queue.cc:1035] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:03.314324 7764 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: TxnStatusTablet state changed. Reason: New leader 4c6a692644a149999bbbd30974cba261. Latest consensus state: current_term: 1 leader_uuid: "4c6a692644a149999bbbd30974cba261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.314627 7764 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:39:03.316444 7772 mvcc.cc:205] Tried to move back new op lower bound from 7317720036579483648 to 7317720036450050048. Current Snapshot: MvccSnapshot[applied={T|T < 7317720036579483648}]
I20260812 16:39:03.318387 7798 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c6a692644a149999bbbd30974cba261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.318648 7798 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:39:03.324851 7767 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: TxnStatusTablet state changed. Reason: New leader 4c6a692644a149999bbbd30974cba261. Latest consensus state: current_term: 1 leader_uuid: "4c6a692644a149999bbbd30974cba261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.325181 7767 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:39:03.325860 7764 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c6a692644a149999bbbd30974cba261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.326121 7764 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 5b0c0624822744d2a899ce29094fc5b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:39:03.332005 7770 mvcc.cc:205] Tried to move back new op lower bound from 7317720036579483648 to 7317720036450050048. Current Snapshot: MvccSnapshot[applied={T|T < 7317720036579483648}]
I20260812 16:39:03.332387 7767 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c6a692644a149999bbbd30974cba261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.332655 7767 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 4cc5b9bbf5704a09ad1f65c0265987ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 16:39:03.341117 7762 tablet_replica.cc:439] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c6a692644a149999bbbd30974cba261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } } peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } }
I20260812 16:39:03.341526 7762 tablet_replica.cc:442] T 5e0a958d1dc349248ca0b2c51273678f P 4c6a692644a149999bbbd30974cba261: This TxnStatusTablet replica's current role is: LEADER
I20260812 16:39:03.353466 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.345644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:44509 (local address 127.31.61.129:43999)
0812 16:39:03.345812 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.345818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.346033 (+ 215us) server_negotiation.cc:407] Connection header received
0812 16:39:03.346397 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.346403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.346496 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.346565 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:39:03.347426 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.348626 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.350783 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.351500 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.351647 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.352363 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:39:03.352544 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:39:03.352726 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.352956 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.353056 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 16:39:03.353524 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.345180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:43999 (local address 127.31.61.130:44509)
0812 16:39:03.345919 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:03.345945 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:03.346181 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:03.346790 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:03.346798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:03.346812 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:39:03.347211 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.347229 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.348784 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.348790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.350614 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.350631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.351749 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.351756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.351959 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.352038 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:39:03.352752 (+ 714us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:39:03.352767 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 16:39:03.352923 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 16:39:03.353086 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":942,"threads_started":1}
I20260812 16:39:03.364915 7676 raft_consensus.cc:1261] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Refusing update from remote peer 4cc5b9bbf5704a09ad1f65c0265987ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:03.364974 7396 raft_consensus.cc:1261] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Refusing update from remote peer 4cc5b9bbf5704a09ad1f65c0265987ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:03.365582 7767 consensus_queue.cc:1035] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:03.366163 7763 consensus_queue.cc:1035] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:03.418432 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.406877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:54619 (local address 127.31.61.130:41711)
0812 16:39:03.407069 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.407079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.407097 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:39:03.407167 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.407173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.407251 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.407357 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:39:03.412421 (+ 5064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.413649 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.415992 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.416678 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.416782 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.417523 (+ 741us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:39:03.417709 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:39:03.417803 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.418025 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.418116 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:39:03.419332 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.405449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:41711 (local address 127.31.61.129:54619)
0812 16:39:03.406254 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:03.406275 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:03.406420 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:03.411758 (+ 5338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:03.411764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:03.411781 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 16:39:03.412194 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.412207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.413863 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.413869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.415794 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:03.415813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.417007 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.417015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:03.417211 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.417300 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 16:39:03.417836 (+ 536us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 16:39:03.417849 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 16:39:03.418734 (+ 885us) client_negotiation.cc:240] Negotiation successful
0812 16:39:03.418893 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":651,"threads_started":1}
I20260812 16:39:03.467972 7230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:03.468276 7230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:03.472123 7230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce: Sending DeleteTablet for 3 replicas of tablet 7fe339daeb8c4bae9e9ab9f2a98be24a
I20260812 16:39:03.473038 7371 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe339daeb8c4bae9e9ab9f2a98be24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:03 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:51786
I20260812 16:39:03.473343 7780 tablet_replica.cc:330] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: stopping tablet replica
I20260812 16:39:03.473294 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe339daeb8c4bae9e9ab9f2a98be24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:03 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:40914
I20260812 16:39:03.473542 7780 raft_consensus.cc:2229] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:03.473591 7781 tablet_replica.cc:330] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: stopping tablet replica
I20260812 16:39:03.473748 7780 raft_consensus.cc:2258] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.473783 7781 raft_consensus.cc:2229] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:03.473964 7781 raft_consensus.cc:2258] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.475246 7781 ts_tablet_manager.cc:1905] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.475317 7780 ts_tablet_manager.cc:1905] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.476917 7513 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe339daeb8c4bae9e9ab9f2a98be24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:03 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:33492
I20260812 16:39:03.477254 7782 tablet_replica.cc:330] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: stopping tablet replica
I20260812 16:39:03.477464 7782 raft_consensus.cc:2229] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:39:03.477809 7782 raft_consensus.cc:2258] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.479362 7782 ts_tablet_manager.cc:1905] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.485106 7781 ts_tablet_manager.cc:1918] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:39:03.485332 7781 log.cc:1196] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/7fe339daeb8c4bae9e9ab9f2a98be24a
I20260812 16:39:03.485740 7781 ts_tablet_manager.cc:1939] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 5b0c0624822744d2a899ce29094fc5b7: Deleting consensus metadata
I20260812 16:39:03.486896 7215 catalog_manager.cc:4841] TS 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973): tablet 7fe339daeb8c4bae9e9ab9f2a98be24a (table test-table [id=7fd0aaf6766748f4aab5790b6af1fa50]) successfully deleted
I20260812 16:39:03.493409 7780 ts_tablet_manager.cc:1918] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:39:03.493721 7780 log.cc:1196] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/7fe339daeb8c4bae9e9ab9f2a98be24a
I20260812 16:39:03.494287 7780 ts_tablet_manager.cc:1939] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4c6a692644a149999bbbd30974cba261: Deleting consensus metadata
I20260812 16:39:03.495807 7216 catalog_manager.cc:4841] TS 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999): tablet 7fe339daeb8c4bae9e9ab9f2a98be24a (table test-table [id=7fd0aaf6766748f4aab5790b6af1fa50]) successfully deleted
I20260812 16:39:03.502089 7782 ts_tablet_manager.cc:1918] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:39:03.502396 7782 log.cc:1196] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/7fe339daeb8c4bae9e9ab9f2a98be24a
I20260812 16:39:03.502957 7782 ts_tablet_manager.cc:1939] T 7fe339daeb8c4bae9e9ab9f2a98be24a P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting consensus metadata
I20260812 16:39:03.504405 7215 catalog_manager.cc:4841] TS 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130:41711): tablet 7fe339daeb8c4bae9e9ab9f2a98be24a (table test-table [id=7fd0aaf6766748f4aab5790b6af1fa50]) successfully deleted
Aug 12 16:39:03 dist-test-slave-f7th krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 16:39:03 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:03 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
W20260812 16:39:03.573163 7327 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:03.559103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.31.61.190:38115)
0812 16:39:03.559326 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.559334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.560146 (+ 812us) server_negotiation.cc:407] Connection header received
0812 16:39:03.560368 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.560375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.560458 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.560527 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:03.562451 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.563574 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.565347 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.566008 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.566144 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.572885 (+ 6741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40772: BlockingRecv error: recv got EOF from 127.0.0.1:40772 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 16:39:03 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM
I20260812 16:39:03.594384 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.577208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.31.61.190:38115)
0812 16:39:03.577400 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.577406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.577427 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:39:03.577623 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.577630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.577709 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.577777 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:03.579692 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.580848 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.582672 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.583331 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.583461 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.588086 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:03.588094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:03.588099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:03.588148 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:03.591395 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:03.592172 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:03.592178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:03.592182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:03.592343 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:03.592714 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:03.592719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:03.592722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:03.593092 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:03.593332 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.593659 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.593919 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:39:03.598035 7230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:39:03.598661 7230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:39:03.605614 7371 tablet_service.cc:1467] Processing CreateTablet for tablet 82f7216a99b741e982b1c805bbc3f196 (DEFAULT_TABLE table=test-table [id=e5e5805470824e5bb4d44231923f27ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:03.605775 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 82f7216a99b741e982b1c805bbc3f196 (DEFAULT_TABLE table=test-table [id=e5e5805470824e5bb4d44231923f27ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:03.605757 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 82f7216a99b741e982b1c805bbc3f196 (DEFAULT_TABLE table=test-table [id=e5e5805470824e5bb4d44231923f27ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:03.606081 7371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f7216a99b741e982b1c805bbc3f196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.606176 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f7216a99b741e982b1c805bbc3f196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.606192 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f7216a99b741e982b1c805bbc3f196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.612638 7757 tablet_bootstrap.cc:492] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: Bootstrap starting.
I20260812 16:39:03.612883 7755 tablet_bootstrap.cc:492] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: Bootstrap starting.
I20260812 16:39:03.614209 7757 tablet_bootstrap.cc:654] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.614750 7755 tablet_bootstrap.cc:654] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.616214 7756 tablet_bootstrap.cc:492] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: Bootstrap starting.
I20260812 16:39:03.617455 7756 tablet_bootstrap.cc:654] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.621372 7757 tablet_bootstrap.cc:492] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: No bootstrap required, opened a new log
I20260812 16:39:03.621552 7757 ts_tablet_manager.cc:1397] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20260812 16:39:03.622279 7757 raft_consensus.cc:348] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.622495 7757 raft_consensus.cc:374] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.622511 7755 tablet_bootstrap.cc:492] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: No bootstrap required, opened a new log
I20260812 16:39:03.622583 7757 raft_consensus.cc:729] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc5b9bbf5704a09ad1f65c0265987ab, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.622659 7755 ts_tablet_manager.cc:1397] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260812 16:39:03.623176 7757 consensus_queue.cc:260] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.623626 7755 raft_consensus.cc:348] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.623869 7757 ts_tablet_manager.cc:1428] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 16:39:03.623894 7755 raft_consensus.cc:374] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.623991 7755 raft_consensus.cc:729] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6a692644a149999bbbd30974cba261, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.624459 7755 consensus_queue.cc:260] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.626051 7756 tablet_bootstrap.cc:492] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: No bootstrap required, opened a new log
I20260812 16:39:03.626207 7756 ts_tablet_manager.cc:1397] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260812 16:39:03.626991 7756 raft_consensus.cc:348] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.627239 7756 raft_consensus.cc:374] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.627342 7756 raft_consensus.cc:729] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0c0624822744d2a899ce29094fc5b7, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.627560 7756 consensus_queue.cc:260] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.628573 7755 ts_tablet_manager.cc:1428] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260812 16:39:03.628615 7756 ts_tablet_manager.cc:1428] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 16:39:03.675223 7762 raft_consensus.cc:482] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:39:03.675518 7762 raft_consensus.cc:504] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.676360 7762 leader_election.cc:288] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130:41711), 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973)
I20260812 16:39:03.677096 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f7216a99b741e982b1c805bbc3f196" candidate_uuid: "4c6a692644a149999bbbd30974cba261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7" is_pre_election: true
I20260812 16:39:03.677066 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f7216a99b741e982b1c805bbc3f196" candidate_uuid: "4c6a692644a149999bbbd30974cba261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" is_pre_election: true
I20260812 16:39:03.677330 7676 raft_consensus.cc:2454] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6a692644a149999bbbd30974cba261 in term 0.
I20260812 16:39:03.677338 7539 raft_consensus.cc:2454] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6a692644a149999bbbd30974cba261 in term 0.
I20260812 16:39:03.677779 7320 leader_election.cc:302] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6a692644a149999bbbd30974cba261, 5b0c0624822744d2a899ce29094fc5b7; no voters:
I20260812 16:39:03.678082 7762 raft_consensus.cc:2781] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:39:03.678215 7762 raft_consensus.cc:482] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:39:03.678315 7762 raft_consensus.cc:3037] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.680775 7762 raft_consensus.cc:504] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.681412 7762 leader_election.cc:288] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130:41711), 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973)
I20260812 16:39:03.681998 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f7216a99b741e982b1c805bbc3f196" candidate_uuid: "4c6a692644a149999bbbd30974cba261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab"
I20260812 16:39:03.682045 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f7216a99b741e982b1c805bbc3f196" candidate_uuid: "4c6a692644a149999bbbd30974cba261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7"
I20260812 16:39:03.682240 7539 raft_consensus.cc:3037] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.682245 7676 raft_consensus.cc:3037] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.685109 7676 raft_consensus.cc:2454] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6a692644a149999bbbd30974cba261 in term 1.
I20260812 16:39:03.685109 7539 raft_consensus.cc:2454] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6a692644a149999bbbd30974cba261 in term 1.
I20260812 16:39:03.685554 7320 leader_election.cc:302] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6a692644a149999bbbd30974cba261, 4cc5b9bbf5704a09ad1f65c0265987ab; no voters:
I20260812 16:39:03.685844 7762 raft_consensus.cc:2781] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:03.686146 7762 raft_consensus.cc:686] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 1 LEADER]: Becoming Leader. State: Replica: 4c6a692644a149999bbbd30974cba261, State: Running, Role: LEADER
I20260812 16:39:03.686458 7762 consensus_queue.cc:237] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.689693 7231 catalog_manager.cc:5495] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6a692644a149999bbbd30974cba261 (127.31.61.129). New cstate: current_term: 1 leader_uuid: "4c6a692644a149999bbbd30974cba261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } health_report { overall_health: UNKNOWN } } }
I20260812 16:39:03.709633 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.704634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.31.61.129:43999)
0812 16:39:03.704866 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.704872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.704991 (+ 119us) server_negotiation.cc:407] Connection header received
0812 16:39:03.705182 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.705188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.705258 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.705339 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:39:03.705705 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.706493 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.707874 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.708321 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.708456 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.708887 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:39:03.709017 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:39:03.709100 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.709246 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.709335 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 16:39:03.713567 7676 raft_consensus.cc:1261] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Refusing update from remote peer 4c6a692644a149999bbbd30974cba261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:03.713862 7539 raft_consensus.cc:1261] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Refusing update from remote peer 4c6a692644a149999bbbd30974cba261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:03.714341 7762 consensus_queue.cc:1035] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:03.714635 7805 consensus_queue.cc:1035] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:03.732295 7230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:03.732570 7230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:03.735800 7230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce: Sending DeleteTablet for 3 replicas of tablet 82f7216a99b741e982b1c805bbc3f196
I20260812 16:39:03.736683 7371 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f7216a99b741e982b1c805bbc3f196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:03 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:51786
I20260812 16:39:03.736756 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f7216a99b741e982b1c805bbc3f196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:03 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:40914
I20260812 16:39:03.736863 7513 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f7216a99b741e982b1c805bbc3f196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:03 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:33492
I20260812 16:39:03.737082 7780 tablet_replica.cc:330] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: stopping tablet replica
I20260812 16:39:03.737177 7781 tablet_replica.cc:330] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: stopping tablet replica
I20260812 16:39:03.737269 7780 raft_consensus.cc:2229] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:39:03.737365 7781 raft_consensus.cc:2229] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:03.737597 7781 raft_consensus.cc:2258] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.737787 7780 raft_consensus.cc:2258] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.737888 7782 tablet_replica.cc:330] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: stopping tablet replica
I20260812 16:39:03.738058 7782 raft_consensus.cc:2229] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:03.738305 7782 raft_consensus.cc:2258] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.738848 7780 ts_tablet_manager.cc:1905] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.739405 7781 ts_tablet_manager.cc:1905] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.740315 7230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:39:03.741042 7230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:39:03.745213 7782 ts_tablet_manager.cc:1905] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.747108 7780 ts_tablet_manager.cc:1918] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:39:03.747272 7780 log.cc:1196] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/82f7216a99b741e982b1c805bbc3f196
I20260812 16:39:03.747733 7780 ts_tablet_manager.cc:1939] T 82f7216a99b741e982b1c805bbc3f196 P 4c6a692644a149999bbbd30974cba261: Deleting consensus metadata
I20260812 16:39:03.748919 7216 catalog_manager.cc:4841] TS 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999): tablet 82f7216a99b741e982b1c805bbc3f196 (table test-table [id=e5e5805470824e5bb4d44231923f27ad]) successfully deleted
I20260812 16:39:03.750806 7782 ts_tablet_manager.cc:1918] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:39:03.750866 7781 ts_tablet_manager.cc:1918] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:39:03.750927 7782 log.cc:1196] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/82f7216a99b741e982b1c805bbc3f196
I20260812 16:39:03.751009 7781 log.cc:1196] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/82f7216a99b741e982b1c805bbc3f196
I20260812 16:39:03.751278 7782 ts_tablet_manager.cc:1939] T 82f7216a99b741e982b1c805bbc3f196 P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting consensus metadata
I20260812 16:39:03.751408 7781 ts_tablet_manager.cc:1939] T 82f7216a99b741e982b1c805bbc3f196 P 5b0c0624822744d2a899ce29094fc5b7: Deleting consensus metadata
I20260812 16:39:03.752302 7215 catalog_manager.cc:4841] TS 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130:41711): tablet 82f7216a99b741e982b1c805bbc3f196 (table test-table [id=e5e5805470824e5bb4d44231923f27ad]) successfully deleted
I20260812 16:39:03.752557 7215 catalog_manager.cc:4841] TS 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973): tablet 82f7216a99b741e982b1c805bbc3f196 (table test-table [id=e5e5805470824e5bb4d44231923f27ad]) successfully deleted
I20260812 16:39:03.756583 7371 tablet_service.cc:1467] Processing CreateTablet for tablet 9d36f1a63f974827b2ca65f727bb387a (DEFAULT_TABLE table=test-table [id=947c56c8c1924e569df60486f3d18ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:03.756775 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 9d36f1a63f974827b2ca65f727bb387a (DEFAULT_TABLE table=test-table [id=947c56c8c1924e569df60486f3d18ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:03.756814 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 9d36f1a63f974827b2ca65f727bb387a (DEFAULT_TABLE table=test-table [id=947c56c8c1924e569df60486f3d18ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:03.757045 7371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d36f1a63f974827b2ca65f727bb387a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.757160 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d36f1a63f974827b2ca65f727bb387a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.757177 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d36f1a63f974827b2ca65f727bb387a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:03.763290 7756 tablet_bootstrap.cc:492] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: Bootstrap starting.
I20260812 16:39:03.763506 7757 tablet_bootstrap.cc:492] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: Bootstrap starting.
I20260812 16:39:03.764256 7755 tablet_bootstrap.cc:492] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: Bootstrap starting.
I20260812 16:39:03.764957 7756 tablet_bootstrap.cc:654] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.764993 7757 tablet_bootstrap.cc:654] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.765787 7755 tablet_bootstrap.cc:654] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:03.775391 7757 tablet_bootstrap.cc:492] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: No bootstrap required, opened a new log
I20260812 16:39:03.775565 7757 ts_tablet_manager.cc:1397] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260812 16:39:03.775439 7755 tablet_bootstrap.cc:492] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: No bootstrap required, opened a new log
I20260812 16:39:03.775985 7755 ts_tablet_manager.cc:1397] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260812 16:39:03.776332 7757 raft_consensus.cc:348] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.776543 7757 raft_consensus.cc:374] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.776620 7757 raft_consensus.cc:729] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc5b9bbf5704a09ad1f65c0265987ab, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.776682 7755 raft_consensus.cc:348] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.776932 7755 raft_consensus.cc:374] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.776849 7757 consensus_queue.cc:260] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.777004 7755 raft_consensus.cc:729] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6a692644a149999bbbd30974cba261, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.777365 7757 ts_tablet_manager.cc:1428] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 16:39:03.778055 7756 tablet_bootstrap.cc:492] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: No bootstrap required, opened a new log
I20260812 16:39:03.777953 7755 consensus_queue.cc:260] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.778211 7756 ts_tablet_manager.cc:1397] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260812 16:39:03.778991 7755 ts_tablet_manager.cc:1428] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 16:39:03.778927 7756 raft_consensus.cc:348] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.779189 7756 raft_consensus.cc:374] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:03.779269 7756 raft_consensus.cc:729] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0c0624822744d2a899ce29094fc5b7, State: Initialized, Role: FOLLOWER
I20260812 16:39:03.779500 7756 consensus_queue.cc:260] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.780835 7756 ts_tablet_manager.cc:1428] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 16:39:03.835479 7767 raft_consensus.cc:482] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:39:03.835759 7767 raft_consensus.cc:504] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.836535 7767 leader_election.cc:288] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999), 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973)
I20260812 16:39:03.837181 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d36f1a63f974827b2ca65f727bb387a" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6a692644a149999bbbd30974cba261" is_pre_election: true
I20260812 16:39:03.837193 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d36f1a63f974827b2ca65f727bb387a" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7" is_pre_election: true
I20260812 16:39:03.837450 7676 raft_consensus.cc:2454] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 0.
I20260812 16:39:03.837460 7396 raft_consensus.cc:2454] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 0.
I20260812 16:39:03.837954 7464 leader_election.cc:302] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6a692644a149999bbbd30974cba261, 4cc5b9bbf5704a09ad1f65c0265987ab; no voters:
I20260812 16:39:03.838327 7767 raft_consensus.cc:2781] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:39:03.838433 7767 raft_consensus.cc:482] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:39:03.838542 7767 raft_consensus.cc:3037] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.841624 7767 raft_consensus.cc:504] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.842805 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d36f1a63f974827b2ca65f727bb387a" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0c0624822744d2a899ce29094fc5b7"
I20260812 16:39:03.842880 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d36f1a63f974827b2ca65f727bb387a" candidate_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6a692644a149999bbbd30974cba261"
I20260812 16:39:03.843020 7676 raft_consensus.cc:3037] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.843113 7396 raft_consensus.cc:3037] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:03.846226 7676 raft_consensus.cc:2454] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 1.
I20260812 16:39:03.846611 7396 raft_consensus.cc:2454] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc5b9bbf5704a09ad1f65c0265987ab in term 1.
I20260812 16:39:03.846762 7463 leader_election.cc:302] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc5b9bbf5704a09ad1f65c0265987ab, 5b0c0624822744d2a899ce29094fc5b7; no voters:
I20260812 16:39:03.847057 7763 raft_consensus.cc:2781] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:03.849566 7767 leader_election.cc:288] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [CANDIDATE]: Term 1 election: Requested vote from peers 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999), 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973)
I20260812 16:39:03.850042 7763 raft_consensus.cc:686] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 LEADER]: Becoming Leader. State: Replica: 4cc5b9bbf5704a09ad1f65c0265987ab, State: Running, Role: LEADER
I20260812 16:39:03.850307 7763 consensus_queue.cc:237] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } }
I20260812 16:39:03.856025 7230 catalog_manager.cc:5495] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130). New cstate: current_term: 1 leader_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc5b9bbf5704a09ad1f65c0265987ab" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 41711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 } health_report { overall_health: UNKNOWN } } }
I20260812 16:39:03.895430 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.889237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.31.61.130:41711)
0812 16:39:03.889401 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.889430 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.889703 (+ 273us) server_negotiation.cc:407] Connection header received
0812 16:39:03.889831 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.889835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.889920 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.890006 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:39:03.890470 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.891202 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.893286 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.893941 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.894068 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.894637 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:39:03.894779 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:39:03.894870 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.895045 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.895137 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 16:39:03.906405 7676 raft_consensus.cc:1261] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Refusing update from remote peer 4cc5b9bbf5704a09ad1f65c0265987ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:03.906404 7396 raft_consensus.cc:1261] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Refusing update from remote peer 4cc5b9bbf5704a09ad1f65c0265987ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:03.907017 7763 consensus_queue.cc:1035] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0c0624822744d2a899ce29094fc5b7" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:03.907539 7838 consensus_queue.cc:1035] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6a692644a149999bbbd30974cba261" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:03.912901 7773 mvcc.cc:205] Tried to move back new op lower bound from 7317720039035191296 to 7317720038813380608. Current Snapshot: MvccSnapshot[applied={T|T < 7317720039035191296}]
I20260812 16:39:03.957772 7230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:03.958029 7230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:03.961750 7230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e4436585474a36b26f7ac87802a0ce: Sending DeleteTablet for 3 replicas of tablet 9d36f1a63f974827b2ca65f727bb387a
I20260812 16:39:03.962704 7371 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d36f1a63f974827b2ca65f727bb387a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:03 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:51786
I20260812 16:39:03.962947 7513 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d36f1a63f974827b2ca65f727bb387a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:03 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:33492
I20260812 16:39:03.963008 7780 tablet_replica.cc:330] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: stopping tablet replica
I20260812 16:39:03.963165 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d36f1a63f974827b2ca65f727bb387a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:03 UTC) from {username='oryx', principal='oryx/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:40914
I20260812 16:39:03.963213 7780 raft_consensus.cc:2229] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:03.963238 7782 tablet_replica.cc:330] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: stopping tablet replica
I20260812 16:39:03.963407 7780 raft_consensus.cc:2258] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.963440 7781 tablet_replica.cc:330] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: stopping tablet replica
I20260812 16:39:03.963441 7782 raft_consensus.cc:2229] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:39:03.963650 7781 raft_consensus.cc:2229] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:03.963805 7782 raft_consensus.cc:2258] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.963828 7781 raft_consensus.cc:2258] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:03.965148 7782 ts_tablet_manager.cc:1905] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.965327 7780 ts_tablet_manager.cc:1905] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.965436 7781 ts_tablet_manager.cc:1905] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:03.972146 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 7304
I20260812 16:39:03.972435 7781 ts_tablet_manager.cc:1918] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:39:03.972705 7781 log.cc:1196] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/9d36f1a63f974827b2ca65f727bb387a
I20260812 16:39:03.972870 7780 ts_tablet_manager.cc:1918] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:39:03.973043 7780 log.cc:1196] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/9d36f1a63f974827b2ca65f727bb387a
I20260812 16:39:03.973305 7781 ts_tablet_manager.cc:1939] T 9d36f1a63f974827b2ca65f727bb387a P 5b0c0624822744d2a899ce29094fc5b7: Deleting consensus metadata
I20260812 16:39:03.973465 7780 ts_tablet_manager.cc:1939] T 9d36f1a63f974827b2ca65f727bb387a P 4c6a692644a149999bbbd30974cba261: Deleting consensus metadata
I20260812 16:39:03.974886 7216 catalog_manager.cc:4841] TS 4c6a692644a149999bbbd30974cba261 (127.31.61.129:43999): tablet 9d36f1a63f974827b2ca65f727bb387a (table test-table [id=947c56c8c1924e569df60486f3d18ec3]) successfully deleted
I20260812 16:39:03.975134 7215 catalog_manager.cc:4841] TS 5b0c0624822744d2a899ce29094fc5b7 (127.31.61.131:41973): tablet 9d36f1a63f974827b2ca65f727bb387a (table test-table [id=947c56c8c1924e569df60486f3d18ec3]) successfully deleted
I20260812 16:39:03.976877 7782 ts_tablet_manager.cc:1918] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 16:39:03.977044 7782 log.cc:1196] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/9d36f1a63f974827b2ca65f727bb387a
I20260812 16:39:03.977438 7782 ts_tablet_manager.cc:1939] T 9d36f1a63f974827b2ca65f727bb387a P 4cc5b9bbf5704a09ad1f65c0265987ab: Deleting consensus metadata
I20260812 16:39:03.978453 7215 catalog_manager.cc:4841] TS 4cc5b9bbf5704a09ad1f65c0265987ab (127.31.61.130:41711): tablet 9d36f1a63f974827b2ca65f727bb387a (table test-table [id=947c56c8c1924e569df60486f3d18ec3]) successfully deleted
Aug 12 16:39:03 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.129@KRBTEST.COM
I20260812 16:39:03.992010 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:03.975872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.31.61.129:43999)
0812 16:39:03.976069 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:03.976076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:03.976097 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:39:03.976154 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:03.976159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:03.976235 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:03.976332 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:03.977896 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.979037 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.980283 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:03.980945 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:03.981068 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:03.985331 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:03.985348 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:03.985365 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:03.985413 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:03.988909 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:03.989752 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:03.989769 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:03.989772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:03.989957 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:03.990408 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:03.990412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:03.990414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:03.990806 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:03.991052 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:03.991398 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 16:39:03.991666 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 16:39:03.992702 7436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:04.166457 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 7304
I20260812 16:39:04.188144 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 7446
Aug 12 16:39:04 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.130@KRBTEST.COM
I20260812 16:39:04.203939 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:04.188882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.31.61.130:41711)
0812 16:39:04.189124 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:04.189128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:04.189147 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:39:04.189202 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:04.189205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:04.189267 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:04.189362 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:04.190551 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:04.191259 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:04.192781 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:04.193220 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:04.193364 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:04.197589 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:04.197607 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:04.197624 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:04.197667 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:04.200749 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:04.201525 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:04.201535 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:04.201537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:04.201683 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:04.202075 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:04.202079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:04.202081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:04.202640 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:04.202903 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:04.203422 (+ 519us) server_negotiation.cc:299] Negotiation successful
0812 16:39:04.203686 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 16:39:04.204650 7579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:04.332006 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 7446
I20260812 16:39:04.353048 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 7589
Aug 12 16:39:04 dist-test-slave-f7th krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.61.131@KRBTEST.COM
I20260812 16:39:04.369164 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:04.353779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.31.61.131:41973)
0812 16:39:04.354370 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:04.354375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:04.354394 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:39:04.354448 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:04.354451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:04.354527 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:04.354667 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:04.356048 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:04.356751 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:04.357992 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:04.358407 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:04.358518 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:04.363076 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:04.363094 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:04.363111 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:04.363153 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:04.366067 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:04.366817 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:04.366826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:04.366828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:04.366982 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:04.367349 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:04.367354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:04.367356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:04.367933 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:04.368242 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:04.368580 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 16:39:04.368821 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":290,"threads_started":1}
I20260812 16:39:04.369849 7723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:04.466367 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 7589
I20260812 16:39:04.481834 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 7198
I20260812 16:39:04.483107 7271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:04.598579 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 7198
2026-08-12T16:39:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6473 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:39:04 dist-test-slave-f7th krb5kdc[7854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:39:04 dist-test-slave-f7th krb5kdc[7854](info): set up 2 sockets
Aug 12 16:39:04 dist-test-slave-f7th krb5kdc[7854](info): commencing operation
krb5kdc: starting...
W20260812 16:39:05.673200 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:39:05 dist-test-slave-f7th krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:39:05.912124 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:37339
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:37339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:06.274013 7867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:06.274366 7867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:06.274495 7867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:06.282016 7867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:39:06.282159 7867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:06.282235 7867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:06.282310 7867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:39:06.282375 7867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:39:06.292551 7867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:37339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:37339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:06.293936 7867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:06.295257 7867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:06.307721 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:06.313093 7874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:06.313938 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:06.314881 7867 server_base.cc:1034] running on GCE node
I20260812 16:39:06.315811 7867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:06.317343 7867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:06.318588 7867 hybrid_clock.cc:648] HybridClock initialized: now 1786552746318541 us; error 36 us; skew 500 ppm
I20260812 16:39:06.319056 7867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T16:39:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1786 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:39:06 dist-test-slave-f7th krb5kdc[7882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:39:06 dist-test-slave-f7th krb5kdc[7882](info): set up 2 sockets
Aug 12 16:39:06 dist-test-slave-f7th krb5kdc[7882](info): commencing operation
krb5kdc: starting...
W20260812 16:39:07.424201 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:39:07 dist-test-slave-f7th krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:39:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:39:07.665082 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36529
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:08.000067 7895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:08.000653 7895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:08.000787 7895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:08.008196 7895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:39:08.008327 7895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:08.008419 7895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:08.008496 7895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:39:08.008560 7895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:39:08.018723 7895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:08.020134 7895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:08.021850 7895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:08.033933 7901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:08.034036 7903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:08.034303 7900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:08.035636 7895 server_base.cc:1034] running on GCE node
I20260812 16:39:08.036432 7895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:08.037879 7895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:08.039108 7895 hybrid_clock.cc:648] HybridClock initialized: now 1786552748039051 us; error 30 us; skew 500 ppm
I20260812 16:39:08.039422 7895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:08 dist-test-slave-f7th krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552748, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:08.049564 7895 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:39:08.051204 7895 webserver.cc:460] Webserver started at http://127.31.61.190:33201/ using document root <none> and password file <none>
I20260812 16:39:08.051959 7895 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:08.052073 7895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:08.052345 7895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:08.055380 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "478938c6b10d41a98667b0d838b22df6"
format_stamp: "Formatted at 2026-08-12 16:39:08 on dist-test-slave-f7th"
server_key: "d6807af49e2202a16d1c8727aafe22b5"
server_key_iv: "b3bde62551f183692548322ef65a1e98"
server_key_version: "encryptionkey@0"
I20260812 16:39:08.056212 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "478938c6b10d41a98667b0d838b22df6"
format_stamp: "Formatted at 2026-08-12 16:39:08 on dist-test-slave-f7th"
server_key: "d6807af49e2202a16d1c8727aafe22b5"
server_key_iv: "b3bde62551f183692548322ef65a1e98"
server_key_version: "encryptionkey@0"
I20260812 16:39:08.061697 7895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:39:08.065346 7911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:08.066699 7895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:39:08.066869 7895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "478938c6b10d41a98667b0d838b22df6"
format_stamp: "Formatted at 2026-08-12 16:39:08 on dist-test-slave-f7th"
server_key: "d6807af49e2202a16d1c8727aafe22b5"
server_key_iv: "b3bde62551f183692548322ef65a1e98"
server_key_version: "encryptionkey@0"
I20260812 16:39:08.067098 7895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:08.079668 7895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:08.084349 7895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:08.084645 7895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:08.102452 7895 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36529
I20260812 16:39:08.102505 7962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36529 every 8 connection(s)
I20260812 16:39:08.103922 7895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:39:08.107754 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:08.108143 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 7895
I20260812 16:39:08.108326 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:39:08.108968 31990 external_mini_cluster.cc:1428] Setting key fcaa50deb408288b4736ad0d80d4089f
I20260812 16:39:08.121968 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: Bootstrap starting.
Aug 12 16:39:08 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:39:08.126015 7963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:08.127133 7963 log.cc:824] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:08.130646 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: No bootstrap required, opened a new log
I20260812 16:39:08.134083 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:08.110855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.31.61.190:36529)
0812 16:39:08.111701 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:08.111713 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:08.111759 (+ 46us) server_negotiation.cc:407] Connection header received
0812 16:39:08.113638 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:08.113664 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:08.114072 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:08.114398 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:08.116440 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:08.118010 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:08.119897 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:08.120663 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:08.120824 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:08.125720 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:08.125730 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:08.125746 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:08.125792 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:08.129240 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:08.130221 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:08.130228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:08.130235 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:08.130411 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:08.130878 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:08.130884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:08.130888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:08.131548 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:08.131876 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:08.132432 (+ 556us) server_negotiation.cc:299] Negotiation successful
0812 16:39:08.132995 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":350,"threads_started":1}
I20260812 16:39:08.138849 7963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } }
I20260812 16:39:08.139241 7963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:08.139338 7963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478938c6b10d41a98667b0d838b22df6, State: Initialized, Role: FOLLOWER
I20260812 16:39:08.139956 7963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } }
I20260812 16:39:08.140184 7963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:39:08.140309 7963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:39:08.140472 7963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:08.142757 7963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } }
I20260812 16:39:08.143280 7963 leader_election.cc:302] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 478938c6b10d41a98667b0d838b22df6; no voters:
I20260812 16:39:08.143951 7963 leader_election.cc:288] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:39:08.144079 7969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:08.144771 7969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 1 LEADER]: Becoming Leader. State: Replica: 478938c6b10d41a98667b0d838b22df6, State: Running, Role: LEADER
I20260812 16:39:08.145308 7969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } }
I20260812 16:39:08.146167 7963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:39:08.147979 7971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 478938c6b10d41a98667b0d838b22df6. Latest consensus state: current_term: 1 leader_uuid: "478938c6b10d41a98667b0d838b22df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } } }
I20260812 16:39:08.148120 7970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "478938c6b10d41a98667b0d838b22df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } } }
I20260812 16:39:08.148633 7970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:08.149168 7971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:08.149793 7977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:39:08.154855 7977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:39:08.164535 7977 catalog_manager.cc:1347] Generated new cluster ID: 47afc20a03bb4089b2e6cf5cd0de9a3a
I20260812 16:39:08.164664 7977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:39:08.191707 7977 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:39:08.193709 7977 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:39:08.206825 7977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: Generated new TSK 0
I20260812 16:39:08.208107 7977 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:39:08.278736 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36529
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:08.618257 7990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:08.618575 7990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:08.618705 7990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:08.626631 7990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:08.626777 7990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:08.626942 7990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:39:08.637494 7990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:08.638823 7990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:08.640192 7990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:08.653744 7995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:08.653781 7998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:08.653805 7996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:08.654655 7990 server_base.cc:1034] running on GCE node
I20260812 16:39:08.655256 7990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:08.656279 7990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:08.657486 7990 hybrid_clock.cc:648] HybridClock initialized: now 1786552748657419 us; error 61 us; skew 500 ppm
I20260812 16:39:08.657799 7990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:08 dist-test-slave-f7th krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552748, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:08.660995 7990 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:39:08.662520 7990 webserver.cc:460] Webserver started at http://127.31.61.129:37695/ using document root <none> and password file <none>
I20260812 16:39:08.663199 7990 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:08.663340 7990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:08.663658 7990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:08.666855 7990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "aa830eeffbcf40d0be0b6222eed072f1"
format_stamp: "Formatted at 2026-08-12 16:39:08 on dist-test-slave-f7th"
server_key: "a477ac234d7e26616d6da80893ae82de"
server_key_iv: "80352c23a868d30243edc6e6df14965d"
server_key_version: "encryptionkey@0"
I20260812 16:39:08.667711 7990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "aa830eeffbcf40d0be0b6222eed072f1"
format_stamp: "Formatted at 2026-08-12 16:39:08 on dist-test-slave-f7th"
server_key: "a477ac234d7e26616d6da80893ae82de"
server_key_iv: "80352c23a868d30243edc6e6df14965d"
server_key_version: "encryptionkey@0"
I20260812 16:39:08.673247 7990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:39:08.677232 8005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:08.678885 7990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 16:39:08.679060 7990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "aa830eeffbcf40d0be0b6222eed072f1"
format_stamp: "Formatted at 2026-08-12 16:39:08 on dist-test-slave-f7th"
server_key: "a477ac234d7e26616d6da80893ae82de"
server_key_iv: "80352c23a868d30243edc6e6df14965d"
server_key_version: "encryptionkey@0"
I20260812 16:39:08.679266 7990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:08.704871 7990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:08.709460 7990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:08.709774 7990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:08.710826 7990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:08.712337 7990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:08.712426 7990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:08.712553 7990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:08.712607 7990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:08.749432 7990 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:43097
I20260812 16:39:08.749485 8121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:43097 every 8 connection(s)
I20260812 16:39:08.750795 7990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:39:08.755813 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 7990
I20260812 16:39:08.756011 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:39:08.756613 31990 external_mini_cluster.cc:1428] Setting key 8e5d860967540c4b47478222b984a8f4
Aug 12 16:39:08 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552748, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:39:08.773001 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:08.753613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:58303 (local address 127.31.61.190:36529)
0812 16:39:08.754373 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:08.754379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:08.755144 (+ 765us) server_negotiation.cc:407] Connection header received
0812 16:39:08.757171 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:08.757176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:08.757262 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:08.757359 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:08.759567 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:08.760526 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:08.762209 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:08.762623 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:08.762742 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:08.766927 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:08.766933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:08.766936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:08.766970 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:08.769618 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:08.770471 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:08.770475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:08.770478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:08.770643 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:08.771084 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:08.771089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:08.771092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:08.771412 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:08.771553 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:08.772369 (+ 816us) server_negotiation.cc:299] Negotiation successful
0812 16:39:08.772630 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":271,"threads_started":1}
I20260812 16:39:08.773977 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:08.754108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:58303)
0812 16:39:08.754919 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:08.754968 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:08.756885 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:08.758044 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:08.758056 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:08.758753 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:08.759377 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:08.759397 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:08.760719 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:08.760724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:08.762040 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:08.762056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:08.762867 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:08.762875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:08.763120 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:08.763901 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:08.763930 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:08.766645 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:08.769795 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:08.769806 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:08.769824 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:08.770353 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:08.770751 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:08.770756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:08.770760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:08.770965 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:08.771542 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:08.771551 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:08.771980 (+ 429us) client_negotiation.cc:769] Sending connection context
0812 16:39:08.772248 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 16:39:08.772822 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":317,"threads_started":1}
I20260812 16:39:08.776242 8122 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36529
I20260812 16:39:08.776585 8122 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:08.777303 8122 heartbeater.cc:511] Master 127.31.61.190:36529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
I20260812 16:39:08.779523 7928 ts_manager.cc:194] Registered new tserver with Master: aa830eeffbcf40d0be0b6222eed072f1 (127.31.61.129:43097)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
I20260812 16:39:08.781376 7928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:58303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:39:08.811885 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36529
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:09.147217 8129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:09.147526 8129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:09.147687 8129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:09.155052 8129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:09.155184 8129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:09.155299 8129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:39:09.166440 8129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:09.167796 8129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:09.169087 8129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:09.183743 8134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:09.183743 8135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:09.184038 8137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:09.184608 8129 server_base.cc:1034] running on GCE node
I20260812 16:39:09.185199 8129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:09.186152 8129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:09.187399 8129 hybrid_clock.cc:648] HybridClock initialized: now 1786552749187319 us; error 48 us; skew 500 ppm
I20260812 16:39:09.187748 8129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:09 dist-test-slave-f7th krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552749, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:09.190757 8129 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:39:09.192189 8129 webserver.cc:460] Webserver started at http://127.31.61.130:44213/ using document root <none> and password file <none>
I20260812 16:39:09.192874 8129 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:09.192974 8129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:09.193297 8129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:09.196199 8129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "d7c28afd129145fa95dab63a1821e7e9"
format_stamp: "Formatted at 2026-08-12 16:39:09 on dist-test-slave-f7th"
server_key: "8d74e5cbe6f819060154ac74299ba698"
server_key_iv: "6ab3c5abcfe71dbb24c02b34a4f8231f"
server_key_version: "encryptionkey@0"
I20260812 16:39:09.196943 8129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "d7c28afd129145fa95dab63a1821e7e9"
format_stamp: "Formatted at 2026-08-12 16:39:09 on dist-test-slave-f7th"
server_key: "8d74e5cbe6f819060154ac74299ba698"
server_key_iv: "6ab3c5abcfe71dbb24c02b34a4f8231f"
server_key_version: "encryptionkey@0"
I20260812 16:39:09.202138 8129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:39:09.205780 8144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:09.207068 8129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 16:39:09.207218 8129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "d7c28afd129145fa95dab63a1821e7e9"
format_stamp: "Formatted at 2026-08-12 16:39:09 on dist-test-slave-f7th"
server_key: "8d74e5cbe6f819060154ac74299ba698"
server_key_iv: "6ab3c5abcfe71dbb24c02b34a4f8231f"
server_key_version: "encryptionkey@0"
I20260812 16:39:09.207399 8129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:09.231262 8129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:09.235553 8129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:09.235862 8129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:09.236887 8129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:09.238299 8129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:09.238468 8129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:09.238627 8129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:09.238690 8129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:09.272383 8129 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:37317
I20260812 16:39:09.272454 8258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:37317 every 8 connection(s)
I20260812 16:39:09.273851 8129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:39:09.274544 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 8129
I20260812 16:39:09.274729 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:39:09.275283 31990 external_mini_cluster.cc:1428] Setting key a75ecfe1ccd2332c2b7e865e03b18cb2
Aug 12 16:39:09 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552749, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:39:09.296294 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:09.276471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:43915 (local address 127.31.61.190:36529)
0812 16:39:09.276876 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:09.276880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:09.278196 (+ 1316us) server_negotiation.cc:407] Connection header received
0812 16:39:09.280370 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:09.280376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:09.280489 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:09.280586 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:09.282197 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:09.282958 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:09.284727 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:09.285170 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:09.285282 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:09.289608 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:09.289615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:09.289618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:09.289655 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:09.292777 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:09.293738 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:09.293743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:09.293745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:09.293914 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:09.294393 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:09.294398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:09.294400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:09.294692 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:09.294847 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:09.295668 (+ 821us) server_negotiation.cc:299] Negotiation successful
0812 16:39:09.295930 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":205,"threads_started":1}
I20260812 16:39:09.297282 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:09.277204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:43915)
0812 16:39:09.278047 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:09.278093 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:09.280086 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:09.280846 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:09.280854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:09.281409 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:09.282020 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:09.282039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:09.283124 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:09.283130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:09.284528 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:09.284545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:09.285415 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:09.285421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:09.285760 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:09.286507 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:09.286538 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:09.289301 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:09.293017 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:09.293039 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:09.293056 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:09.293579 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:09.294042 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:09.294046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:09.294049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:09.294270 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:09.294827 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:09.294836 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:09.295219 (+ 383us) client_negotiation.cc:769] Sending connection context
0812 16:39:09.295486 (+ 267us) client_negotiation.cc:240] Negotiation successful
0812 16:39:09.296108 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":351,"threads_started":1}
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
I20260812 16:39:09.299028 8259 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36529
I20260812 16:39:09.299382 8259 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:09.300302 8259 heartbeater.cc:511] Master 127.31.61.190:36529 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
I20260812 16:39:09.302196 7928 ts_manager.cc:194] Registered new tserver with Master: d7c28afd129145fa95dab63a1821e7e9 (127.31.61.130:37317)
I20260812 16:39:09.303290 7928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:43915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:39:09.332489 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36529
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:09.662397 8265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:09.662700 8265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:09.662834 8265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:09.670122 8265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:09.670255 8265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:09.670401 8265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:39:09.680740 8265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:09.682068 8265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:09.683429 8265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:09.696326 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:09.696446 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:09.696426 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:09.701157 8265 server_base.cc:1034] running on GCE node
I20260812 16:39:09.701790 8265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:09.702772 8265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:09.703986 8265 hybrid_clock.cc:648] HybridClock initialized: now 1786552749703930 us; error 43 us; skew 500 ppm
I20260812 16:39:09.704299 8265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:09 dist-test-slave-f7th krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552749, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:09.708307 8265 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:39:09.710315 8265 webserver.cc:460] Webserver started at http://127.31.61.131:36191/ using document root <none> and password file <none>
I20260812 16:39:09.711333 8265 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:09.711476 8265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:09.711933 8265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:09.715924 8265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "4706a0311d614145b55dd9856118bd2d"
format_stamp: "Formatted at 2026-08-12 16:39:09 on dist-test-slave-f7th"
server_key: "a7eff6e36b846a1a43345741b54eb229"
server_key_iv: "535916e142b26f5ed6b1422fd6f5d8b2"
server_key_version: "encryptionkey@0"
I20260812 16:39:09.717027 8265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "4706a0311d614145b55dd9856118bd2d"
format_stamp: "Formatted at 2026-08-12 16:39:09 on dist-test-slave-f7th"
server_key: "a7eff6e36b846a1a43345741b54eb229"
server_key_iv: "535916e142b26f5ed6b1422fd6f5d8b2"
server_key_version: "encryptionkey@0"
I20260812 16:39:09.723933 8265 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260812 16:39:09.728715 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:09.730348 8265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 16:39:09.730532 8265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "4706a0311d614145b55dd9856118bd2d"
format_stamp: "Formatted at 2026-08-12 16:39:09 on dist-test-slave-f7th"
server_key: "a7eff6e36b846a1a43345741b54eb229"
server_key_iv: "535916e142b26f5ed6b1422fd6f5d8b2"
server_key_version: "encryptionkey@0"
I20260812 16:39:09.730737 8265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:09.784986 8122 heartbeater.cc:503] Master 127.31.61.190:36529 was elected leader, sending a full tablet report...
I20260812 16:39:09.794045 8265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:09.798421 8265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:09.798673 8265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:09.799777 8265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:09.801337 8265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:09.801414 8265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:09.801498 8265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:09.801540 8265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:09.834061 8265 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:42565
I20260812 16:39:09.834116 8395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:42565 every 8 connection(s)
I20260812 16:39:09.835435 8265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:39:09.836279 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 8265
I20260812 16:39:09.836445 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:39:09.836830 31990 external_mini_cluster.cc:1428] Setting key 8dc5dcc941ae4030691e7d6b9f649803
Aug 12 16:39:09 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552749, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:39:09.856724 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:09.838226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39881 (local address 127.31.61.190:36529)
0812 16:39:09.838707 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:09.838712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:09.841939 (+ 3227us) server_negotiation.cc:407] Connection header received
0812 16:39:09.841992 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:09.841997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:09.842064 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:09.842143 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:09.843719 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:09.844573 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:09.846205 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:09.846651 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:09.846757 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:09.850645 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:09.850651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:09.850654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:09.850689 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:09.853611 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:09.854350 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:09.854354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:09.854356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:09.854477 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:09.854904 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:09.854908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:09.854910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:09.855193 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:09.855335 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:09.856119 (+ 784us) server_negotiation.cc:299] Negotiation successful
0812 16:39:09.856351 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":240,"threads_started":1}
I20260812 16:39:09.857671 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:09.839032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:39881)
0812 16:39:09.839906 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:09.839951 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:09.841671 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:09.842396 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:09.842405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:09.842943 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:09.843512 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:09.843530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:09.844753 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:09.844757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:09.846051 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:09.846064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:09.846874 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:09.846880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:09.847097 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:09.847917 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:09.847945 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:09.850384 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:09.853755 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:09.853766 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:09.853782 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:09.854226 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:09.854581 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:09.854587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:09.854590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:09.854784 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:09.855318 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:09.855326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:09.855729 (+ 403us) client_negotiation.cc:769] Sending connection context
0812 16:39:09.855981 (+ 252us) client_negotiation.cc:240] Negotiation successful
0812 16:39:09.856555 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":341,"threads_started":1}
I20260812 16:39:09.859021 8396 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36529
I20260812 16:39:09.859315 8396 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:09.860217 8396 heartbeater.cc:511] Master 127.31.61.190:36529 requested a full tablet report, sending...
I20260812 16:39:09.861833 7928 ts_manager.cc:194] Registered new tserver with Master: 4706a0311d614145b55dd9856118bd2d (127.31.61.131:42565)
I20260812 16:39:09.862685 7928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:39881
I20260812 16:39:09.865618 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab.
Entry for principal oryx/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab.
WARNING: no policy specified for oryx/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab.
Entry for principal oryx/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab.
WARNING: no policy specified for oryx/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab.
Entry for principal oryx/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab.
I20260812 16:39:10.015137 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 7990
Aug 12 16:39:10 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:39:10.033674 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:10.016455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.31.61.129:43097)
0812 16:39:10.017147 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:10.017155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:10.017186 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:39:10.017252 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:10.017259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:10.017509 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:10.017662 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:10.019161 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:10.020335 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:10.021811 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:10.022437 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:10.022579 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:10.026995 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:10.027011 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:10.027028 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:10.027075 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:10.030277 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:10.031189 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:10.031197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:10.031200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:10.031376 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:10.031945 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:10.031949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:10.031952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:10.032362 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:10.032679 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:10.032963 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 16:39:10.033184 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":391,"threads_started":1}
I20260812 16:39:10.035034 8117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:10.119150 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 7990
I20260812 16:39:10.133387 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8129
Aug 12 16:39:10 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:39:10.150400 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:10.134300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.31.61.130:37317)
0812 16:39:10.134850 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:10.134858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:10.134883 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:39:10.134947 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:10.134954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:10.135200 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:10.135346 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:10.136674 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:10.137795 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:10.139094 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:10.139677 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:10.139793 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:10.144378 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:10.144391 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:10.144407 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:10.144455 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:10.147525 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:10.148259 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:10.148267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:10.148269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:10.148407 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:10.148773 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:10.148776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:10.148779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:10.149145 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:10.149459 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:10.149772 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 16:39:10.150044 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
I20260812 16:39:10.151899 8254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:10.241624 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 8129
I20260812 16:39:10.256390 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8265
Aug 12 16:39:10 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:39:10.273613 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:10.257321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.31.61.131:42565)
0812 16:39:10.257906 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:10.257914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:10.257954 (+ 40us) server_negotiation.cc:407] Connection header received
0812 16:39:10.258019 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:10.258026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:10.258280 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:10.258467 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:10.259822 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:10.260854 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:10.262182 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:10.262766 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:10.262883 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:10.267615 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:10.267635 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:10.267657 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:10.267702 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:10.270541 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:10.271257 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:10.271264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:10.271267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:10.271428 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:10.271964 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:10.271968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:10.271970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:10.272338 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:10.272669 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:10.272975 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 16:39:10.273220 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
I20260812 16:39:10.274978 8391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:10.361658 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 8265
I20260812 16:39:10.375727 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 7895
I20260812 16:39:10.376981 7958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:10.481312 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 7895
I20260812 16:39:10.500962 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36529
--webserver_interface=127.31.61.190
--webserver_port=33201
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:10.842025 8415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:10.842674 8415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:10.842828 8415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:10.850306 8415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:39:10.850435 8415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:10.850499 8415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:10.850580 8415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:39:10.850639 8415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:39:10.861049 8415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:10.862495 8415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:10.863883 8415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:10.875851 8422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:10.875862 8423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:10.876539 8425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:10.876977 8415 server_base.cc:1034] running on GCE node
I20260812 16:39:10.877794 8415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:10.879334 8415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:10.880570 8415 hybrid_clock.cc:648] HybridClock initialized: now 1786552750880512 us; error 43 us; skew 500 ppm
I20260812 16:39:10.880954 8415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:10 dist-test-slave-f7th krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552750, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:10.884202 8415 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:39:10.885826 8415 webserver.cc:460] Webserver started at http://127.31.61.190:33201/ using document root <none> and password file <none>
I20260812 16:39:10.886571 8415 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:10.886703 8415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:10.892345 8415 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:39:10.895206 8432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:10.896617 8415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:39:10.896780 8415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "478938c6b10d41a98667b0d838b22df6"
format_stamp: "Formatted at 2026-08-12 16:39:08 on dist-test-slave-f7th"
server_key: "d6807af49e2202a16d1c8727aafe22b5"
server_key_iv: "b3bde62551f183692548322ef65a1e98"
server_key_version: "encryptionkey@0"
I20260812 16:39:10.897785 8415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:10.912599 8415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:10.916512 8415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:10.916762 8415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:10.934962 8415 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36529
I20260812 16:39:10.934989 8484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36529 every 8 connection(s)
I20260812 16:39:10.936589 8415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:39:10.944072 8485 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:39:10.944350 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 8415
I20260812 16:39:10.945133 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:43097
--local_ip_for_outbound_sockets=127.31.61.129
--tserver_master_addrs=127.31.61.190:36529
--webserver_port=37695
--webserver_interface=127.31.61.129
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.61.129
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:39:10.947824 8485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: Bootstrap starting.
I20260812 16:39:10.973312 8485 log.cc:824] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:10.985433 8485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:39:10.986119 8485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: Bootstrap complete.
I20260812 16:39:10.999280 8485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } }
I20260812 16:39:11.000635 8485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478938c6b10d41a98667b0d838b22df6, State: Initialized, Role: FOLLOWER
I20260812 16:39:11.001497 8485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } }
I20260812 16:39:11.001783 8485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:39:11.001912 8485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:39:11.002100 8485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:39:11.008036 8485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } }
I20260812 16:39:11.008842 8485 leader_election.cc:302] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 478938c6b10d41a98667b0d838b22df6; no voters:
I20260812 16:39:11.009594 8485 leader_election.cc:288] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 16:39:11.009791 8489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:39:11.010954 8489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [term 2 LEADER]: Becoming Leader. State: Replica: 478938c6b10d41a98667b0d838b22df6, State: Running, Role: LEADER
I20260812 16:39:11.011699 8489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } }
I20260812 16:39:11.012113 8485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:39:11.014206 8490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "478938c6b10d41a98667b0d838b22df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } } }
I20260812 16:39:11.014184 8491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 478938c6b10d41a98667b0d838b22df6. Latest consensus state: current_term: 2 leader_uuid: "478938c6b10d41a98667b0d838b22df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478938c6b10d41a98667b0d838b22df6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36529 } } }
I20260812 16:39:11.014609 8491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:11.014617 8490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:11.015524 8495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:39:11.033918 8495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:39:11.035682 8495 catalog_manager.cc:1259] Loaded cluster ID: 47afc20a03bb4089b2e6cf5cd0de9a3a
I20260812 16:39:11.035854 8495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:39:11.040201 8495 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:39:11.041950 8495 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 478938c6b10d41a98667b0d838b22df6: Loaded TSK: 0
I20260812 16:39:11.043674 8495 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:39:11.327438 8487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:11.327788 8487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:11.327931 8487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:11.336647 8487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:11.336776 8487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:11.336930 8487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:39:11.347486 8487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.129.keytab
--principal=oryx/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:43097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=37695
--tserver_master_addrs=127.31.61.190:36529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:11.348894 8487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:11.350186 8487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:11.366134 8512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:11.366266 8515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:11.366297 8513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:11.367503 8487 server_base.cc:1034] running on GCE node
I20260812 16:39:11.368197 8487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:11.369192 8487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:11.370427 8487 hybrid_clock.cc:648] HybridClock initialized: now 1786552751370381 us; error 47 us; skew 500 ppm
I20260812 16:39:11.370785 8487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:11 dist-test-slave-f7th krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552751, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:11.374122 8487 init.cc:377] Logged in from keytab as oryx/127.31.61.129@KRBTEST.COM (short username oryx)
I20260812 16:39:11.375787 8487 webserver.cc:460] Webserver started at http://127.31.61.129:37695/ using document root <none> and password file <none>
I20260812 16:39:11.376487 8487 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:11.376622 8487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:11.382447 8487 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260812 16:39:11.385617 8522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:11.387454 8487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 16:39:11.387671 8487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "aa830eeffbcf40d0be0b6222eed072f1"
format_stamp: "Formatted at 2026-08-12 16:39:08 on dist-test-slave-f7th"
server_key: "a477ac234d7e26616d6da80893ae82de"
server_key_iv: "80352c23a868d30243edc6e6df14965d"
server_key_version: "encryptionkey@0"
I20260812 16:39:11.388696 8487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:11.410677 8487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:11.415099 8487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:11.415354 8487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:11.416400 8487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:11.417807 8487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:11.417904 8487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:11.418018 8487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:11.418093 8487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:11.448205 8487 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:43097
I20260812 16:39:11.448263 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:43097 every 8 connection(s)
I20260812 16:39:11.449620 8487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:39:11.454008 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 8487
I20260812 16:39:11.454576 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:37317
--local_ip_for_outbound_sockets=127.31.61.130
--tserver_master_addrs=127.31.61.190:36529
--webserver_port=44213
--webserver_interface=127.31.61.130
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.61.130
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:39:11 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:11 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
W20260812 16:39:11.796748 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:11.797051 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:11.797176 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:11.805374 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:11.805496 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:11.805646 8644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:39:11.816134 8644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.130.keytab
--principal=oryx/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:37317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=44213
--tserver_master_addrs=127.31.61.190:36529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:11.817463 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:11.819075 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:11.835917 8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:11.836018 8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:11.835907 8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:11.836862 8644 server_base.cc:1034] running on GCE node
I20260812 16:39:11.837495 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:11.838465 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:11.839723 8644 hybrid_clock.cc:648] HybridClock initialized: now 1786552751839647 us; error 69 us; skew 500 ppm
I20260812 16:39:11.840098 8644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:11 dist-test-slave-f7th krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552751, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:11.843439 8644 init.cc:377] Logged in from keytab as oryx/127.31.61.130@KRBTEST.COM (short username oryx)
I20260812 16:39:11.845070 8644 webserver.cc:460] Webserver started at http://127.31.61.130:44213/ using document root <none> and password file <none>
I20260812 16:39:11.845839 8644 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:11.845978 8644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:11.851584 8644 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:39:11.854621 8661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:11.856123 8644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:39:11.856274 8644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "d7c28afd129145fa95dab63a1821e7e9"
format_stamp: "Formatted at 2026-08-12 16:39:09 on dist-test-slave-f7th"
server_key: "8d74e5cbe6f819060154ac74299ba698"
server_key_iv: "6ab3c5abcfe71dbb24c02b34a4f8231f"
server_key_version: "encryptionkey@0"
I20260812 16:39:11.857234 8644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:11.871918 8644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:11.876398 8644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:11.876655 8644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:11.877688 8644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:11.879127 8644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:11.879216 8644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:11.879321 8644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:11.879390 8644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:11.919553 8644 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:37317
I20260812 16:39:11.919616 8775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:37317 every 8 connection(s)
I20260812 16:39:11.921159 8644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:39:11.927654 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 8644
I20260812 16:39:11.928498 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:42565
--local_ip_for_outbound_sockets=127.31.61.131
--tserver_master_addrs=127.31.61.190:36529
--webserver_port=36191
--webserver_interface=127.31.61.131
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.61.131
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:39:11 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:11 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
W20260812 16:39:12.280930 8780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:12.281263 8780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:12.281399 8780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:12.288978 8780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:12.289120 8780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:12.289281 8780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:39:12.299481 8780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/krb5kdc/oryx_127.31.61.131.keytab
--principal=oryx/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:42565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=36191
--tserver_master_addrs=127.31.61.190:36529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:12.300889 8780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:12.302292 8780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:12.323354 8789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:12.323392 8788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:12.331588 8792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:12.332899 8780 server_base.cc:1034] running on GCE node
I20260812 16:39:12.333595 8780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:12.334688 8780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:12.335908 8780 hybrid_clock.cc:648] HybridClock initialized: now 1786552752335848 us; error 50 us; skew 500 ppm
I20260812 16:39:12.336246 8780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:12 dist-test-slave-f7th krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552752, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:12.339372 8780 init.cc:377] Logged in from keytab as oryx/127.31.61.131@KRBTEST.COM (short username oryx)
I20260812 16:39:12.341302 8780 webserver.cc:460] Webserver started at http://127.31.61.131:36191/ using document root <none> and password file <none>
I20260812 16:39:12.342054 8780 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:12.342190 8780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:12.347945 8780 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:39:12.350767 8798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:12.352245 8780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:39:12.352415 8780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "4706a0311d614145b55dd9856118bd2d"
format_stamp: "Formatted at 2026-08-12 16:39:09 on dist-test-slave-f7th"
server_key: "a7eff6e36b846a1a43345741b54eb229"
server_key_iv: "535916e142b26f5ed6b1422fd6f5d8b2"
server_key_version: "encryptionkey@0"
I20260812 16:39:12.353400 8780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:12.385283 8780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:12.389811 8780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:12.390090 8780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:12.391139 8780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:12.392759 8780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:12.392855 8780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:12.392964 8780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:12.393047 8780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:12.436427 8780 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:42565
I20260812 16:39:12.436473 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:42565 every 8 connection(s)
I20260812 16:39:12.437885 8780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:39:12.443748 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 8780
Aug 12 16:39:12 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:12 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:12.470265 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:12.445160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.31.61.190:36529)
0812 16:39:12.448597 (+ 3437us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:12.448603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:12.448618 (+ 15us) server_negotiation.cc:407] Connection header received
0812 16:39:12.448669 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:12.448674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:12.448739 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:12.448838 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:12.450982 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.452301 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.454157 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.454732 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.454835 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:12.458898 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:12.458913 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:12.458941 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:12.459017 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:12.463544 (+ 4527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:12.464673 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:12.464686 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:12.464698 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:12.464954 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:12.465459 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:12.465465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:12.465469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:12.466240 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:12.466609 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:12.468421 (+ 1812us) server_negotiation.cc:299] Negotiation successful
0812 16:39:12.469007 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3315,"thread_start_us":1679,"threads_started":1}
W20260812 16:39:12.800590 8643 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.338s user 0.006s sys 0.006s
W20260812 16:39:12.802654 8642 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:11.452623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:59417 (local address 127.31.61.190:36529)
0812 16:39:11.453435 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:11.453452 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:11.453676 (+ 224us) server_negotiation.cc:407] Connection header received
0812 16:39:11.455895 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:11.455908 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:11.456246 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:11.456567 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:11.458104 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:11.459338 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:11.460900 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:11.461558 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:11.461741 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:12.802233 (+1340492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:59417: BlockingRecv error: recv got EOF from 127.31.61.129:59417 (error 108)
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":303,"threads_started":1}
I20260812 16:39:12.803182 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:11.452658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:59417)
0812 16:39:11.453499 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:11.453540 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:11.455330 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:11.456769 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:11.456781 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:11.457371 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:11.457902 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:11.457920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:11.459506 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:11.459510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:11.460758 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:11.460771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:11.461809 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:11.461817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:11.462064 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:11.462682 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:11.462711 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:12.801796 (+1339085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
W20260812 16:39:12.804423 8640 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:36529 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.61.190:36529: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Aug 12 16:39:12 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:12 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:12.819474 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:12.805319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:47205)
0812 16:39:12.805645 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:12.805669 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:12.805804 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:12.806203 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:12.806209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:12.806739 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:12.806984 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:12.807005 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.808613 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.808619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:12.811289 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:12.811311 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.812456 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.812463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:12.812671 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:12.813504 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:12.813532 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:12.819095 (+ 5563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183}
W20260812 16:39:12.819787 8642 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:12.805414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:47205 (local address 127.31.61.190:36529)
0812 16:39:12.805738 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:12.805746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:12.805775 (+ 29us) server_negotiation.cc:407] Connection header received
0812 16:39:12.805932 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:12.805939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:12.806032 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:12.806164 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:12.807150 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.808442 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.811467 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.812251 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.812391 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:12.819446 (+ 7055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:47205: BlockingRecv error: recv got EOF from 127.31.61.129:47205 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Aug 12 16:39:12 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:12 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:12.835398 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:12.820754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:44287)
0812 16:39:12.820961 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:12.820986 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:12.821132 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:12.821939 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:12.821945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:12.822449 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:12.822699 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:12.822713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.825149 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.825156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:12.827294 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:12.827320 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.828483 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.828491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:12.828815 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:12.829686 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:12.829714 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:12.835039 (+ 5325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260812 16:39:12.836645 8642 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:12.821358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:44287 (local address 127.31.61.190:36529)
0812 16:39:12.821567 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:12.821574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:12.821598 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:39:12.821669 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:12.821675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:12.821766 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:12.821890 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:12.823154 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.824782 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.827506 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.828303 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.828456 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:12.836314 (+ 7858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:44287: BlockingRecv error: recv got EOF from 127.31.61.129:44287 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 16:39:12.837692 8640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:39:13.107658 8779 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.174s user 0.001s sys 0.015s
W20260812 16:39:13.109108 8778 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:11.924057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:38295 (local address 127.31.61.190:36529)
0812 16:39:11.924511 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:11.924516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:11.925906 (+ 1390us) server_negotiation.cc:407] Connection header received
0812 16:39:11.927818 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:11.927825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:11.927912 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:11.928015 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:11.929639 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:11.930385 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:11.932156 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:11.932562 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:11.932683 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.108721 (+1176038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:38295: BlockingRecv error: recv got EOF from 127.31.61.130:38295 (error 108)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":216,"threads_started":1}
I20260812 16:39:13.109266 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:11.924848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:38295)
0812 16:39:11.925745 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:11.925790 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:11.927507 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:11.928286 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:11.928297 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:11.928897 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:11.929456 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:11.929473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:11.930545 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:11.930549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:11.931997 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:11.932011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:11.932808 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:11.932816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:11.933036 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:11.933672 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:11.933695 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:13.108385 (+1174690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":342,"threads_started":1}
W20260812 16:39:13.110368 8776 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:36529 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.61.190:36529: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:13.127452 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:13.111138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:48855)
0812 16:39:13.111323 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:13.111343 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:13.111468 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:13.111825 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:13.111830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:13.112196 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:13.112359 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.112367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.113581 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.113587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.114540 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.114552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.115264 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.115271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.115424 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.122404 (+ 6980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:13.122452 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:13.127065 (+ 4613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 16:39:13.127497 8778 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:13.111226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:48855 (local address 127.31.61.190:36529)
0812 16:39:13.111410 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:13.111416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:13.111434 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:39:13.111586 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:13.111591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:13.111692 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:13.111776 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:13.112497 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.113422 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.114687 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.115097 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.115204 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.127301 (+ 12097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:48855: BlockingRecv error: recv got EOF from 127.31.61.130:48855 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:13.139561 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:13.128530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:35289)
0812 16:39:13.128791 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:13.128817 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:13.128956 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:13.129365 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:13.129370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:13.129899 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:13.130162 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.130177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.131441 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.131447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.132485 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.132499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.133463 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.133470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.133704 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.134411 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:13.134437 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:13.139262 (+ 4825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260812 16:39:13.139845 8778 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:13.128754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:35289 (local address 127.31.61.190:36529)
0812 16:39:13.128903 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:13.128908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:13.128925 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:39:13.129137 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:13.129142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:13.129226 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:13.129315 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:13.130348 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.131282 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.132634 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.133249 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.133410 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.139535 (+ 6125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:35289: BlockingRecv error: recv got EOF from 127.31.61.130:35289 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 16:39:13.140128 8776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:39:13.500831 8918 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.044s user 0.004s sys 0.016s
W20260812 16:39:13.502156 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:12.442337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:40117 (local address 127.31.61.190:36529)
0812 16:39:12.447099 (+ 4762us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:12.447106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:12.447127 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:39:12.447198 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:12.447204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:12.447292 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:12.447402 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:12.450752 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.452261 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.454634 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.455253 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.455364 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.501875 (+1046511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:40117: BlockingRecv error: recv got EOF from 127.31.61.131:40117 (error 108)
Metrics: {"server-negotiator.queue_time_us":4626,"thread_start_us":6018,"threads_started":1}
I20260812 16:39:13.502740 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:12.441671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:40117)
0812 16:39:12.442547 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:12.442604 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:12.444878 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:12.448827 (+ 3949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:12.448839 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:12.449650 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:12.450509 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:12.450538 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.452455 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.452461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:12.454451 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:12.454467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:12.455714 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:12.455722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:12.456035 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:12.457128 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:12.457160 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:13.501706 (+1044546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":656,"threads_started":1}
W20260812 16:39:13.503661 8916 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:36529 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.61.190:36529: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:13.512398 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:13.504329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:57673)
0812 16:39:13.504519 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:13.504536 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:13.504626 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:13.504906 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:13.504909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:13.505238 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:13.505405 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.505415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.506449 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.506454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.507436 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.507449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.508136 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.508140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.508260 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.508841 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:13.508858 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:13.512116 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 16:39:13.512593 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:13.504384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:57673 (local address 127.31.61.190:36529)
0812 16:39:13.504549 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:13.504554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:13.504574 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:39:13.504713 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:13.504717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:13.504777 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:13.504855 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:13.505565 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.506280 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.507560 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.508000 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.508092 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.512321 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:57673: BlockingRecv error: recv got EOF from 127.31.61.131:57673 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:13.521106 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:13.513115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:60517)
0812 16:39:13.513275 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:13.513295 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:13.513385 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:13.513723 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:13.513728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:13.514125 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:13.514291 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.514299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.515209 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.515213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.516249 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.516260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.516994 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.516999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.517156 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.517686 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:13.517702 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:13.520888 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 16:39:13.521378 8916 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:39:13.521391 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:13.513174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:60517 (local address 127.31.61.190:36529)
0812 16:39:13.513425 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:13.513431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:13.513449 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:39:13.513499 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:13.513504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:13.513595 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:13.513692 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:13.514415 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.515106 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.516370 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.516821 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.516917 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.521127 (+ 4210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:60517: BlockingRecv error: recv got EOF from 127.31.61.131:60517 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:13 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:13.847908 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:13.838448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:41547)
0812 16:39:13.838994 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:13.839010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:13.839125 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:13.839508 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:13.839512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:13.839989 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:13.840146 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.840155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.841223 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.841227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.842239 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:13.842253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.843050 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.843056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:13.843190 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.843795 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:13.843816 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:13.847657 (+ 3841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":290,"threads_started":1}
W20260812 16:39:13.848174 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:13.838552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:41547 (local address 127.31.61.190:36529)
0812 16:39:13.838738 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:13.838745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:13.839076 (+ 331us) server_negotiation.cc:407] Connection header received
0812 16:39:13.839235 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:13.839239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:13.839335 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:13.839437 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:13.840300 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.841064 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.842396 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:13.842838 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:13.842945 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:13.847948 (+ 5003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:41547: BlockingRecv error: recv got EOF from 127.31.61.129:41547 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 16:39:14 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:14 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:14.150352 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:14.140966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:41901)
0812 16:39:14.141518 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:14.141535 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:14.141657 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:14.142109 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:14.142114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:14.142701 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:14.142857 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:14.142867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.144019 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.144023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:14.144992 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:14.145004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.145784 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.145789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:14.145912 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:14.146480 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:14.146498 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:14.150112 (+ 3614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":304,"threads_started":1}
W20260812 16:39:14.150595 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:14.140972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:41901 (local address 127.31.61.190:36529)
0812 16:39:14.141187 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:14.141194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:14.141640 (+ 446us) server_negotiation.cc:407] Connection header received
0812 16:39:14.141754 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:14.141759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:14.141919 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:14.142037 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:14.143009 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.143851 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.145146 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.145600 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.145706 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:14.150347 (+ 4641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:41901: BlockingRecv error: recv got EOF from 127.31.61.130:41901 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 12 16:39:14 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:14 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:14.531015 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:14.521958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:52481)
0812 16:39:14.522510 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:14.522527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:14.522653 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:14.522980 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:14.522993 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:14.523421 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:14.523576 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:14.523586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.524702 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.524708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:14.525725 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:14.525740 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.526496 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.526503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:14.526674 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:14.527244 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:14.527262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:14.530772 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":333,"threads_started":1}
W20260812 16:39:14.531215 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:14.521973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:52481 (local address 127.31.61.190:36529)
0812 16:39:14.522160 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:14.522166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:14.522607 (+ 441us) server_negotiation.cc:407] Connection header received
0812 16:39:14.522774 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:14.522778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:14.522844 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:14.522927 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:14.523745 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.524465 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.525860 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.526304 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.526415 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:14.530992 (+ 4577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:52481: BlockingRecv error: recv got EOF from 127.31.61.131:52481 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 16:39:14 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:14 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:14.858752 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:14.848834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:54357)
0812 16:39:14.849312 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:14.849328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:14.849442 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:14.849747 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:14.849750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:14.850169 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:14.850355 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:14.850363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.851405 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.851409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:14.852440 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:14.852453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.853242 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.853247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:14.853565 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:14.854163 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:14.854183 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:14.858468 (+ 4285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":277,"threads_started":1}
W20260812 16:39:14.859055 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:14.848917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:54357 (local address 127.31.61.190:36529)
0812 16:39:14.849141 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:14.849147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:14.849398 (+ 251us) server_negotiation.cc:407] Connection header received
0812 16:39:14.849561 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:14.849566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:14.849633 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:14.849715 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:14.850517 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.851262 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.852600 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:14.853054 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:14.853164 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:14.858795 (+ 5631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:54357: BlockingRecv error: recv got EOF from 127.31.61.129:54357 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 16:39:15 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:15 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:15.161177 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:15.151346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:57525)
0812 16:39:15.151903 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:15.151920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:15.152040 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:15.152458 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:15.152462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:15.152875 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:15.153082 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:15.153093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.154180 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.154187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:15.155144 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:15.155158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.156054 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.156060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:15.156250 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:15.156904 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:15.156928 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:15.160856 (+ 3928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":335,"threads_started":1}
W20260812 16:39:15.161262 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:15.151488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:57525 (local address 127.31.61.190:36529)
0812 16:39:15.151700 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:15.151705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:15.151986 (+ 281us) server_negotiation.cc:407] Connection header received
0812 16:39:15.152191 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:15.152198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:15.152269 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:15.152371 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:15.153240 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.153982 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.155293 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.155842 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.156001 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:15.161068 (+ 5067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:57525: BlockingRecv error: recv got EOF from 127.31.61.130:57525 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 16:39:15 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:15 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:15.541186 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:15.531947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:41111)
0812 16:39:15.532394 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:15.532410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:15.532554 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:15.532888 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:15.532893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:15.533292 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:15.533500 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:15.533510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.534545 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.534550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:15.535525 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:15.535539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.536255 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.536260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:15.536422 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:15.536989 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:15.537008 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:15.540768 (+ 3760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":249,"threads_started":1}
W20260812 16:39:15.541337 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:15.532101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:41111 (local address 127.31.61.190:36529)
0812 16:39:15.532348 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:15.532353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:15.532506 (+ 153us) server_negotiation.cc:407] Connection header received
0812 16:39:15.532643 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:15.532647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:15.532726 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:15.532810 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:15.533643 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.534405 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.535690 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.536108 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.536206 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:15.541077 (+ 4871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:41111: BlockingRecv error: recv got EOF from 127.31.61.131:41111 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 12 16:39:15 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:15 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:15.868705 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:15.859791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:38295)
0812 16:39:15.860328 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:15.860343 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:15.860475 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:15.860821 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:15.860826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:15.861291 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:15.861464 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:15.861474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.862584 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.862588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:15.863504 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:15.863516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.864233 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.864239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:15.864408 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:15.864951 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:15.864968 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:15.868386 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":302,"threads_started":1}
W20260812 16:39:15.868878 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:15.859879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:38295 (local address 127.31.61.190:36529)
0812 16:39:15.860096 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:15.860103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:15.860429 (+ 326us) server_negotiation.cc:407] Connection header received
0812 16:39:15.860611 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:15.860617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:15.860695 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:15.860786 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:15.861610 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.862446 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.863664 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:15.864094 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:15.864203 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:15.868646 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:38295: BlockingRecv error: recv got EOF from 127.31.61.129:38295 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 16:39:16 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:16 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:16.171588 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:16.162264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:48609)
0812 16:39:16.162779 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:16.162796 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:16.162921 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:16.163359 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:16.163363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:16.163780 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:16.163961 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:16.163971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.165078 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.165083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:16.166017 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:16.166030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.166801 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.166806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:16.166968 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:16.167618 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:16.167639 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:16.171278 (+ 3639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":274,"threads_started":1}
W20260812 16:39:16.171830 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:16.162418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:48609 (local address 127.31.61.190:36529)
0812 16:39:16.162683 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:16.162689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:16.162923 (+ 234us) server_negotiation.cc:407] Connection header received
0812 16:39:16.163019 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:16.163023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:16.163098 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:16.163193 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:16.164129 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.164904 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.166177 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.166640 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.166741 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:16.171537 (+ 4796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:48609: BlockingRecv error: recv got EOF from 127.31.61.130:48609 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Aug 12 16:39:16 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:16 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:16.551728 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:16.542300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:39503)
0812 16:39:16.542816 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:16.542834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:16.542958 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:16.543392 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:16.543398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:16.543824 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:16.544054 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:16.544066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.545341 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.545345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:16.546428 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:16.546444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.547205 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.547210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:16.547369 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:16.548066 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:16.548085 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:16.551397 (+ 3312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":268,"threads_started":1}
W20260812 16:39:16.551908 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:16.542491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39503 (local address 127.31.61.190:36529)
0812 16:39:16.542681 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:16.542686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:16.542943 (+ 257us) server_negotiation.cc:407] Connection header received
0812 16:39:16.543090 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:16.543096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:16.543196 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:16.543311 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:16.544203 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.545197 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.546585 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.547040 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.547142 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:16.551677 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:39503: BlockingRecv error: recv got EOF from 127.31.61.131:39503 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 16:39:16 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:16 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:16.878680 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:16.869840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:41657)
0812 16:39:16.870362 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:16.870402 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:16.870544 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:16.870867 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:16.870871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:16.871280 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:16.871456 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:16.871466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.872532 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.872537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:16.873544 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:16.873556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.874219 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.874225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:16.874402 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:16.874954 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:16.874974 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:16.878294 (+ 3320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":280,"threads_started":1}
W20260812 16:39:16.878753 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:16.869944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:41657 (local address 127.31.61.190:36529)
0812 16:39:16.870181 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:16.870188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:16.870495 (+ 307us) server_negotiation.cc:407] Connection header received
0812 16:39:16.870663 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:16.870668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:16.870732 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:16.870816 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:16.871628 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.872396 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.873683 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:16.874085 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:16.874191 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:16.878522 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:41657: BlockingRecv error: recv got EOF from 127.31.61.129:41657 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 12 16:39:17 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:17 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:17.181610 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:17.172816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:37757)
0812 16:39:17.173406 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:17.173425 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:17.173570 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:17.173912 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:17.173916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:17.174271 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:17.174439 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:17.174448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.175410 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.175414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:17.176423 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:17.176435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.177181 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.177185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:17.177303 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:17.177865 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:17.177883 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:17.181321 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":354,"threads_started":1}
W20260812 16:39:17.181748 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:17.172922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:37757 (local address 127.31.61.190:36529)
0812 16:39:17.173107 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:17.173114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:17.173540 (+ 426us) server_negotiation.cc:407] Connection header received
0812 16:39:17.173698 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:17.173703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:17.173784 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:17.173868 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:17.174563 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.175295 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.176564 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.177072 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.177172 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:17.181539 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:37757: BlockingRecv error: recv got EOF from 127.31.61.130:37757 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 16:39:17 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:17 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:17.562119 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:17.552811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:37961)
0812 16:39:17.553379 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:17.553398 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:17.553540 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:17.553959 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:17.553964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:17.554473 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:17.554650 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:17.554660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.555714 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.555718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:17.556735 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:17.556751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.557511 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.557516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:17.557655 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:17.558248 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:17.558265 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:17.561839 (+ 3574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":319,"threads_started":1}
W20260812 16:39:17.562316 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:17.552899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:37961 (local address 127.31.61.190:36529)
0812 16:39:17.553080 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:17.553086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:17.553522 (+ 436us) server_negotiation.cc:407] Connection header received
0812 16:39:17.553667 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:17.553673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:17.553788 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:17.553896 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:17.554809 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.555560 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.556896 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.557321 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.557461 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:17.562076 (+ 4615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:37961: BlockingRecv error: recv got EOF from 127.31.61.131:37961 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 12 16:39:17 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:17 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:17.888834 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:17.879862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:33413)
0812 16:39:17.880472 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:17.880488 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:17.880631 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:17.880954 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:17.880959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:17.881394 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:17.881559 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:17.881594 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.882592 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.882598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:17.883533 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:17.883546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.884251 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.884257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:17.884423 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:17.885009 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:17.885029 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:17.888464 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":370,"threads_started":1}
W20260812 16:39:17.888965 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:17.879998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:33413 (local address 127.31.61.190:36529)
0812 16:39:17.880197 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:17.880204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:17.880575 (+ 371us) server_negotiation.cc:407] Connection header received
0812 16:39:17.880747 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:17.880752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:17.880819 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:17.880900 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:17.881732 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.882445 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.883706 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:17.884108 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:17.884227 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:17.888720 (+ 4493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:33413: BlockingRecv error: recv got EOF from 127.31.61.129:33413 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 16:39:18 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:18 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:18.195406 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:18.182509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:54793)
0812 16:39:18.183019 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:18.183037 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:18.183175 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:18.183503 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:18.183508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:18.183995 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:18.184150 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:18.184159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.185244 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.185249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:18.186182 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:18.186194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.186898 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.186904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:18.187074 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:18.191144 (+ 4070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:18.191173 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:18.195061 (+ 3888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":245,"threads_started":1}
W20260812 16:39:18.195616 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:18.182766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:54793 (local address 127.31.61.190:36529)
0812 16:39:18.182978 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:18.182985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:18.183112 (+ 127us) server_negotiation.cc:407] Connection header received
0812 16:39:18.183287 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:18.183291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:18.183353 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:18.183440 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:18.184296 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.185107 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.186324 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.186760 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.186870 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:18.195321 (+ 8451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:54793: BlockingRecv error: recv got EOF from 127.31.61.130:54793 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 16:39:18 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:18 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:18.572048 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:18.563062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:53077)
0812 16:39:18.563566 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:18.563583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:18.563735 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:18.564084 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:18.564089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:18.564587 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:18.564736 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:18.564745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.565761 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.565766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:18.566702 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:18.566714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.567399 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.567405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:18.567573 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:18.568148 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:18.568166 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:18.571820 (+ 3654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":247,"threads_started":1}
W20260812 16:39:18.572335 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:18.563231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:53077 (local address 127.31.61.190:36529)
0812 16:39:18.563448 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:18.563454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:18.563681 (+ 227us) server_negotiation.cc:407] Connection header received
0812 16:39:18.563860 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:18.563866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:18.563943 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:18.564034 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:18.564881 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.565625 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.566841 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.567262 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.567369 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:18.572098 (+ 4729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:53077: BlockingRecv error: recv got EOF from 127.31.61.131:53077 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 16:39:18 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:18 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:18.898341 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:18.889848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:34139)
0812 16:39:18.890352 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:18.890376 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:18.890509 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:18.890821 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:18.890826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:18.891229 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:18.891406 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:18.891415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.892510 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.892515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:18.893463 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:18.893476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.894187 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.894193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:18.894357 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:18.894911 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:18.894928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:18.898100 (+ 3172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":288,"threads_started":1}
W20260812 16:39:18.898502 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:18.889953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:34139 (local address 127.31.61.190:36529)
0812 16:39:18.890186 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:18.890193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:18.890459 (+ 266us) server_negotiation.cc:407] Connection header received
0812 16:39:18.890619 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:18.890625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:18.890689 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:18.890774 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:18.891546 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.892384 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.893595 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:18.894043 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:18.894149 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:18.898300 (+ 4151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:34139: BlockingRecv error: recv got EOF from 127.31.61.129:34139 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 16:39:19 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:19 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:19.205361 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:19.196497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:58483)
0812 16:39:19.197022 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:19.197043 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:19.197196 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:19.197540 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:19.197544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:19.197947 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:19.198136 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:19.198146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.199138 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.199143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:19.200139 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:19.200152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.200875 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.200882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:19.201043 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:19.201631 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:19.201648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:19.205130 (+ 3482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
W20260812 16:39:19.205600 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:19.196839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:58483 (local address 127.31.61.190:36529)
0812 16:39:19.197040 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:19.197045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:19.197110 (+ 65us) server_negotiation.cc:407] Connection header received
0812 16:39:19.197311 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:19.197315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:19.197384 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:19.197475 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:19.198255 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.199004 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.200273 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.200711 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.200808 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:19.205348 (+ 4540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:58483: BlockingRecv error: recv got EOF from 127.31.61.130:58483 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 12 16:39:19 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:19 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:19.582419 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:19.572966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:43893)
0812 16:39:19.573506 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:19.573522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:19.573654 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:19.573983 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:19.573989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:19.574424 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:19.574601 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:19.574610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.575958 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.575965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:19.576932 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:19.576945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.577624 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.577630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:19.577810 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:19.578412 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:19.578435 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:19.582180 (+ 3745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":308,"threads_started":1}
W20260812 16:39:19.582733 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:19.573084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:43893 (local address 127.31.61.190:36529)
0812 16:39:19.573334 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:19.573340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:19.573610 (+ 270us) server_negotiation.cc:407] Connection header received
0812 16:39:19.573768 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:19.573773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:19.573841 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:19.573928 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:19.574767 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.575772 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.577075 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.577459 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.577579 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:19.582440 (+ 4861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:43893: BlockingRecv error: recv got EOF from 127.31.61.131:43893 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 12 16:39:19 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:19 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:19.909051 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:19.899398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:50035)
0812 16:39:19.900097 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:19.900113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:19.900260 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:19.900732 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:19.900737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:19.901237 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:19.901467 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:19.901479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.902624 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.902629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:19.903675 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:19.903688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.904569 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.904575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:19.904738 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:19.905296 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:19.905314 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:19.908695 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":340,"threads_started":1}
W20260812 16:39:19.909143 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:19.899664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:50035 (local address 127.31.61.190:36529)
0812 16:39:19.899902 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:19.899908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:19.900209 (+ 301us) server_negotiation.cc:407] Connection header received
0812 16:39:19.900457 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:19.900463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:19.900555 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:19.900672 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:19.901614 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.902474 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.903822 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:19.904381 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:19.904516 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:19.908921 (+ 4405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:50035: BlockingRecv error: recv got EOF from 127.31.61.129:50035 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 16:39:20 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:20 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:20.216147 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:20.206313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:57465)
0812 16:39:20.206890 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:20.206911 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:20.207064 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:20.207540 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:20.207545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:20.207981 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:20.208173 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:20.208183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.209257 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.209262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:20.210269 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:20.210285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.211060 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.211065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:20.211213 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:20.211970 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:20.211992 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:20.215887 (+ 3895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":316,"threads_started":1}
W20260812 16:39:20.216334 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:20.206637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:57465 (local address 127.31.61.190:36529)
0812 16:39:20.206844 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:20.206851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:20.206992 (+ 141us) server_negotiation.cc:407] Connection header received
0812 16:39:20.207191 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:20.207197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:20.207279 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:20.207374 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:20.208315 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.209071 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.210419 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.210875 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.210976 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:20.216103 (+ 5127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:57465: BlockingRecv error: recv got EOF from 127.31.61.130:57465 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 16:39:20 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:20 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:20.592885 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:20.583548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:35403)
0812 16:39:20.584253 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:20.584272 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:20.584410 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:20.584860 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:20.584882 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:20.585294 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:20.585481 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:20.585491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.586512 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.586517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:20.587482 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:20.587494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.588196 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.588202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:20.588327 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:20.588931 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:20.588950 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:20.592644 (+ 3694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":307,"threads_started":1}
W20260812 16:39:20.593086 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:20.583665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:35403 (local address 127.31.61.190:36529)
0812 16:39:20.583876 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:20.583880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:20.584398 (+ 518us) server_negotiation.cc:407] Connection header received
0812 16:39:20.584532 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:20.584536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:20.584608 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:20.584767 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:20.585658 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.586380 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.587641 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.588036 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.588132 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:20.592839 (+ 4707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:35403: BlockingRecv error: recv got EOF from 127.31.61.131:35403 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 12 16:39:20 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:20 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:20.919904 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:20.910159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:42201)
0812 16:39:20.910794 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:20.910811 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:20.910953 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:20.911368 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:20.911373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:20.911832 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:20.912011 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:20.912021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.913164 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.913168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:20.914143 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:20.914156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.915087 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.915096 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:20.915259 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:20.915979 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:20.916004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:20.919632 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":289,"threads_started":1}
W20260812 16:39:20.920156 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:20.910430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:42201 (local address 127.31.61.190:36529)
0812 16:39:20.910697 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:20.910704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:20.910900 (+ 196us) server_negotiation.cc:407] Connection header received
0812 16:39:20.911143 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:20.911149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:20.911224 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:20.911312 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:20.912164 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.913015 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.914289 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:20.914844 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:20.914968 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:20.919897 (+ 4929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:42201: BlockingRecv error: recv got EOF from 127.31.61.129:42201 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 12 16:39:21 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:21 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:21.227051 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:21.217139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:36627)
0812 16:39:21.217861 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:21.217879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:21.218007 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:21.218354 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:21.218358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:21.218915 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:21.219112 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:21.219123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.220416 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.220420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:21.221409 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:21.221423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.222235 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.222241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:21.222386 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:21.222999 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:21.223024 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:21.226782 (+ 3758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":457,"threads_started":1}
W20260812 16:39:21.227377 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:21.217236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:36627 (local address 127.31.61.190:36529)
0812 16:39:21.217473 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:21.217479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:21.217981 (+ 502us) server_negotiation.cc:407] Connection header received
0812 16:39:21.218108 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:21.218112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:21.218197 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:21.218290 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:21.219295 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.220258 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.221591 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.222029 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.222133 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:21.227116 (+ 4983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:36627: BlockingRecv error: recv got EOF from 127.31.61.130:36627 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 16:39:21 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:21 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:21.604127 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:21.593869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:58753)
0812 16:39:21.594536 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:21.594558 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:21.594704 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:21.595041 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:21.595045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:21.595629 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:21.595898 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:21.595914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.597157 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.597162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:21.598111 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:21.598125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.598902 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.598909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:21.599063 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:21.600006 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:21.600026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:21.603870 (+ 3844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":310,"threads_started":1}
W20260812 16:39:21.604537 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:21.594018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:58753 (local address 127.31.61.190:36529)
0812 16:39:21.594270 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:21.594276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:21.594670 (+ 394us) server_negotiation.cc:407] Connection header received
0812 16:39:21.594830 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:21.594836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:21.594906 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:21.595001 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:21.596091 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.596989 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.598269 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.598715 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.598829 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:21.604208 (+ 5379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:58753: BlockingRecv error: recv got EOF from 127.31.61.131:58753 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 12 16:39:21 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:21 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:21.930231 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:21.921056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:37199)
0812 16:39:21.921638 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:21.921659 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:21.921797 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:21.922118 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:21.922123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:21.922654 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:21.922850 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:21.922862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.923973 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.923977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:21.924928 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:21.924940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.925647 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.925651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:21.925784 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:21.926338 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:21.926360 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:21.929884 (+ 3524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":268,"threads_started":1}
W20260812 16:39:21.930371 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:21.921190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:37199 (local address 127.31.61.190:36529)
0812 16:39:21.921390 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:21.921396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:21.921766 (+ 370us) server_negotiation.cc:407] Connection header received
0812 16:39:21.921911 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:21.921915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:21.921981 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:21.922070 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:21.923023 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.923824 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.925070 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:21.925508 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:21.925604 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:21.930137 (+ 4533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:37199: BlockingRecv error: recv got EOF from 127.31.61.129:37199 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 16:39:22 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:22 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:22.239048 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:22.228260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:52215)
0812 16:39:22.229019 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:22.229052 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:22.229288 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:22.229782 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:22.229786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:22.230344 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:22.230509 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:22.230519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.231908 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.231913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:22.232950 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:22.232965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.233777 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.233783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:22.233964 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:22.234580 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:22.234601 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:22.238708 (+ 4107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":431,"threads_started":1}
W20260812 16:39:22.239279 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:22.228424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:52215 (local address 127.31.61.190:36529)
0812 16:39:22.228686 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:22.228695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:22.229210 (+ 515us) server_negotiation.cc:407] Connection header received
0812 16:39:22.229457 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:22.229461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:22.229533 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:22.229623 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:22.230771 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.231737 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.233126 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.233618 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.233737 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:22.239002 (+ 5265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:52215: BlockingRecv error: recv got EOF from 127.31.61.130:52215 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 12 16:39:22 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:22 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:22.615001 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:22.605359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:39253)
0812 16:39:22.605985 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:22.606003 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:22.606158 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:22.606568 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:22.606574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:22.607019 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:22.607192 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:22.607203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.608395 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.608402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:22.609425 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:22.609439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.610333 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.610340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:22.610516 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:22.611127 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:22.611146 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:22.614747 (+ 3601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":301,"threads_started":1}
W20260812 16:39:22.615389 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:22.605551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39253 (local address 127.31.61.190:36529)
0812 16:39:22.605834 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:22.605851 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:22.606094 (+ 243us) server_negotiation.cc:407] Connection header received
0812 16:39:22.606302 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:22.606307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:22.606390 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:22.606515 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:22.607364 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.608249 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.609653 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.610163 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.610291 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:22.615011 (+ 4720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:39253: BlockingRecv error: recv got EOF from 127.31.61.131:39253 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 12 16:39:22 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:22 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:22.941720 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:22.931474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:45901)
0812 16:39:22.932069 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:22.932086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:22.932208 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:22.932537 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:22.932540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:22.932940 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:22.933135 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:22.933145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.934189 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.934193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:22.935170 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:22.935182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.936349 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.936356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:22.936553 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:22.937188 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:22.937208 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:22.941396 (+ 4188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":308,"threads_started":1}
W20260812 16:39:22.941943 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:22.931590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:45901 (local address 127.31.61.190:36529)
0812 16:39:22.931788 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:22.931794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:22.932186 (+ 392us) server_negotiation.cc:407] Connection header received
0812 16:39:22.932315 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:22.932320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:22.932390 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:22.932476 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:22.933286 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.934062 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.935313 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:22.936107 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:22.936256 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:22.941649 (+ 5393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:45901: BlockingRecv error: recv got EOF from 127.31.61.129:45901 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 12 16:39:23 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:23 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:23.253402 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:23.240238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:40795)
0812 16:39:23.240822 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:23.240844 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:23.240994 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:23.241468 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:23.241478 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:23.241971 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:23.242156 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:23.242167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.243193 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.243198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:23.244275 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:23.244289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.244978 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.244983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:23.245114 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:23.249180 (+ 4066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:23.249201 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:23.253101 (+ 3900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":301,"threads_started":1}
W20260812 16:39:23.253638 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:23.240445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:40795 (local address 127.31.61.190:36529)
0812 16:39:23.240667 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:23.240674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:23.240973 (+ 299us) server_negotiation.cc:407] Connection header received
0812 16:39:23.241112 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:23.241118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:23.241203 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:23.241295 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:23.242297 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.243041 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.244429 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.244848 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.244948 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:23.253357 (+ 8409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:40795: BlockingRecv error: recv got EOF from 127.31.61.130:40795 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 12 16:39:23 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:23 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:23.624981 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:23.615897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:40641)
0812 16:39:23.616390 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:23.616407 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:23.616570 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:23.616898 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:23.616902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:23.617277 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:23.617440 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:23.617449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.618517 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.618521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:23.619728 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:23.619744 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.620523 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.620529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:23.620657 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:23.621256 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:23.621275 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:23.624720 (+ 3445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
W20260812 16:39:23.625290 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:23.616107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:40641 (local address 127.31.61.190:36529)
0812 16:39:23.616360 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:23.616364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:23.616505 (+ 141us) server_negotiation.cc:407] Connection header received
0812 16:39:23.616679 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:23.616683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:23.616752 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:23.616838 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:23.617612 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.618357 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.619879 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.620324 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.620461 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:23.624978 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:40641: BlockingRecv error: recv got EOF from 127.31.61.131:40641 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Aug 12 16:39:23 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:23 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:23.952440 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:23.942870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:45487)
0812 16:39:23.943350 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:23.943367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:23.943495 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:23.943829 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:23.943832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:23.944260 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:23.944454 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:23.944465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.945528 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.945532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:23.946495 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:23.946508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.947278 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.947283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:23.947428 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:23.948041 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:23.948059 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:23.952108 (+ 4049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":250,"threads_started":1}
W20260812 16:39:23.952562 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:23.942970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:45487 (local address 127.31.61.190:36529)
0812 16:39:23.943147 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:23.943153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:23.943476 (+ 323us) server_negotiation.cc:407] Connection header received
0812 16:39:23.943627 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:23.943633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:23.943709 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:23.943794 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:23.944606 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.945371 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.946651 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:23.947135 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:23.947233 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:23.952333 (+ 5100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:45487: BlockingRecv error: recv got EOF from 127.31.61.129:45487 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 16:39:24 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:24 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:24.264477 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:24.254486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:46807)
0812 16:39:24.254975 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:24.254992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:24.255108 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:24.255396 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:24.255400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:24.255880 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:24.256091 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:24.256101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.257187 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.257192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:24.258204 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:24.258238 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.258969 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.258975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:24.259146 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:24.259802 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:24.259821 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:24.264197 (+ 4376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
W20260812 16:39:24.264716 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:24.254574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:46807 (local address 127.31.61.190:36529)
0812 16:39:24.254737 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:24.254742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:24.255089 (+ 347us) server_negotiation.cc:407] Connection header received
0812 16:39:24.255204 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:24.255208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:24.255278 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:24.255363 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:24.256231 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.257032 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.258366 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.258813 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.258924 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:24.264454 (+ 5530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:46807: BlockingRecv error: recv got EOF from 127.31.61.130:46807 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 16:39:24 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:24 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:24.635664 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:24.626081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:49879)
0812 16:39:24.626647 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:24.626666 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:24.626788 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:24.627161 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:24.627165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:24.627583 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:24.627831 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:24.627842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.629128 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.629134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:24.630445 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:24.630461 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.631243 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.631248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:24.631383 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:24.632076 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:24.632095 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:24.635397 (+ 3302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":379,"threads_started":1}
W20260812 16:39:24.635844 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:24.626267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:49879 (local address 127.31.61.190:36529)
0812 16:39:24.626506 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:24.626512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:24.626769 (+ 257us) server_negotiation.cc:407] Connection header received
0812 16:39:24.626886 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:24.626890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:24.626973 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:24.627077 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:24.628040 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.628943 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.630609 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.631084 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.631190 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:24.635617 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:49879: BlockingRecv error: recv got EOF from 127.31.61.131:49879 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 16:39:24 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:24 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:24.963415 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:24.953551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:52297)
0812 16:39:24.954097 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:24.954114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:24.954236 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:24.954601 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:24.954605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:24.954996 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:24.955167 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:24.955178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.956262 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.956266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:24.957360 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:24.957378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.958182 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.958187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:24.958380 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:24.959055 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:24.959079 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:24.963033 (+ 3954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":301,"threads_started":1}
W20260812 16:39:24.963500 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:24.953647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:52297 (local address 127.31.61.190:36529)
0812 16:39:24.953804 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:24.953808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:24.954197 (+ 389us) server_negotiation.cc:407] Connection header received
0812 16:39:24.954345 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:24.954349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:24.954417 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:24.954493 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:24.955301 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.956113 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.957528 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:24.958026 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:24.958161 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:24.963267 (+ 5106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:52297: BlockingRecv error: recv got EOF from 127.31.61.129:52297 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 16:39:25 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:25 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:25.275628 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:25.265616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:58687)
0812 16:39:25.266171 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:25.266188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:25.266325 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:25.266680 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:25.266683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:25.267132 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:25.267308 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:25.267317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.268647 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.268653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:25.269610 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:25.269625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.270516 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.270521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:25.270669 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:25.271303 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:25.271323 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:25.275273 (+ 3950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":273,"threads_started":1}
W20260812 16:39:25.275807 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:25.265706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:58687 (local address 127.31.61.190:36529)
0812 16:39:25.265921 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:25.265927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:25.266289 (+ 362us) server_negotiation.cc:407] Connection header received
0812 16:39:25.266469 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:25.266473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:25.266539 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:25.266621 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:25.267481 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.268259 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.269784 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.270281 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.270441 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:25.275534 (+ 5093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:58687: BlockingRecv error: recv got EOF from 127.31.61.130:58687 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 16:39:25 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:25 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:25.647032 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:25.636812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:39275)
0812 16:39:25.637532 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:25.637549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:25.637672 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:25.638043 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:25.638047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:25.638654 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:25.638841 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:25.638852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.640072 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.640077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:25.641080 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:25.641108 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.642023 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.642029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:25.642172 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:25.643110 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:25.643136 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:25.646772 (+ 3636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":353,"threads_started":1}
W20260812 16:39:25.647221 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:25.636973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39275 (local address 127.31.61.190:36529)
0812 16:39:25.637217 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:25.637224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:25.637672 (+ 448us) server_negotiation.cc:407] Connection header received
0812 16:39:25.637772 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:25.637776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:25.637846 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:25.637953 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:25.639026 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.639889 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.641255 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.641810 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.641930 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:25.647001 (+ 5071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:39275: BlockingRecv error: recv got EOF from 127.31.61.131:39275 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 12 16:39:25 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:25 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:25.974313 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:25.964592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:40405)
0812 16:39:25.965165 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:25.965183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:25.965340 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:25.965780 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:25.965784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:25.966297 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:25.966507 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:25.966519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.967711 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.967716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:25.968642 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:25.968655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.969489 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.969495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:25.969637 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:25.970227 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:25.970249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:25.974057 (+ 3808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":337,"threads_started":1}
W20260812 16:39:25.974642 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:25.964718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:40405 (local address 127.31.61.190:36529)
0812 16:39:25.964984 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:25.964990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:25.965278 (+ 288us) server_negotiation.cc:407] Connection header received
0812 16:39:25.965523 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:25.965528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:25.965603 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:25.965721 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:25.966661 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.967546 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.968791 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:25.969213 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:25.969315 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:25.974338 (+ 5023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:40405: BlockingRecv error: recv got EOF from 127.31.61.129:40405 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 12 16:39:26 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:26 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:26.285889 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:26.276683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:39411)
0812 16:39:26.277182 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:26.277199 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:26.277324 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:26.277647 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:26.277650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:26.278077 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:26.278232 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:26.278241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.279277 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.279280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:26.280256 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:26.280270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.281082 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.281087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:26.281236 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:26.281909 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:26.281972 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:26.285648 (+ 3676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
W20260812 16:39:26.286183 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:26.276883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:39411 (local address 127.31.61.190:36529)
0812 16:39:26.277113 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:26.277119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:26.277285 (+ 166us) server_negotiation.cc:407] Connection header received
0812 16:39:26.277449 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:26.277453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:26.277521 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:26.277605 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:26.278375 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.279165 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.280412 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.280912 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.281021 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:26.285917 (+ 4896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:39411: BlockingRecv error: recv got EOF from 127.31.61.130:39411 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 12 16:39:26 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:26 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.131@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:26.657472 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:26.648069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.131:36029)
0812 16:39:26.648603 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:26.648622 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:26.648777 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:26.649155 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:26.649161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:26.649646 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:26.649810 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:26.649821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.650934 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.650942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:26.651962 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:26.651976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.652720 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.652726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:26.652913 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:26.653568 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:26.653588 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:26.657231 (+ 3643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":297,"threads_started":1}
W20260812 16:39:26.657698 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:26.648173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:36029 (local address 127.31.61.190:36529)
0812 16:39:26.648452 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:26.648460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:26.648723 (+ 263us) server_negotiation.cc:407] Connection header received
0812 16:39:26.648927 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:26.648933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:26.649006 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:26.649105 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:26.649985 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.650774 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.652120 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.652553 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.652674 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:26.657464 (+ 4790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.131:36029: BlockingRecv error: recv got EOF from 127.31.61.131:36029 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 12 16:39:26 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:26 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.129@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:26.984907 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:26.975577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.129:51549)
0812 16:39:26.976154 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:26.976172 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:26.976292 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:26.976662 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:26.976665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:26.977051 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:26.977234 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:26.977244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.978592 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.978597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:26.979790 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:26.979806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.980518 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.980522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:26.980649 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:26.981215 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:26.981232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:26.984662 (+ 3430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":302,"threads_started":1}
W20260812 16:39:26.985127 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:26.975706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:51549 (local address 127.31.61.190:36529)
0812 16:39:26.975931 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:26.975936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:26.976246 (+ 310us) server_negotiation.cc:407] Connection header received
0812 16:39:26.976445 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:26.976450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:26.976523 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:26.976623 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:26.977405 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.978414 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.979971 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:26.980394 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:26.980504 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:26.984858 (+ 4354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.129:51549: BlockingRecv error: recv got EOF from 127.31.61.129:51549 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Aug 12 16:39:27 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
Aug 12 16:39:27 dist-test-slave-f7th krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.61.130@KRBTEST.COM for oryx/127.31.61.190@KRBTEST.COM, Server not found in Kerberos database
I20260812 16:39:27.297015 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:27.287266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36529 (local address 127.31.61.130:43419)
0812 16:39:27.287938 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:27.287954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:27.288084 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:27.288403 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:27.288416 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:27.288803 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:27.288994 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:27.289004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.290022 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.290027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:27.290946 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:27.290959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.291670 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.291676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:27.291901 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:27.292623 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:27.292642 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:27.296575 (+ 3933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.61.190:36529: Server oryx/127.31.61.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":237,"threads_started":1}
W20260812 16:39:27.297170 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:27.287628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:43419 (local address 127.31.61.190:36529)
0812 16:39:27.287798 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:27.287803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:27.288037 (+ 234us) server_negotiation.cc:407] Connection header received
0812 16:39:27.288194 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:27.288200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:27.288264 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:27.288345 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:27.289139 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.289867 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.291083 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.291500 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.291638 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:27.296919 (+ 5281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.61.130:43419: BlockingRecv error: recv got EOF from 127.31.61.130:43419 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 16:39:27.449272 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8487
I20260812 16:39:27.463218 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:27.450736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.31.61.129:43097)
0812 16:39:27.451417 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:27.451425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:27.451459 (+ 34us) server_negotiation.cc:407] Connection header received
0812 16:39:27.451528 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:27.451536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:27.451843 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:27.452010 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:27.453913 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.455060 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.456342 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.456983 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.457100 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:27.459781 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:27.459791 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:27.459806 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:27.459855 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:27.462548 (+ 2693us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.61.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 16:39:27.462955 (+ 407us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53516: Request ticket server kudu/127.31.61.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":345,"threads_started":1}
W20260812 16:39:27.463327 8966 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53516: Request ticket server kudu/127.31.61.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 16:39:27.463347 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8487: Not authorized: Client connection negotiation failed: client connection to 127.31.61.129:43097: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.61.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 16:39:27.463562 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 8487
I20260812 16:39:27.478858 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8644
I20260812 16:39:27.491647 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:27.480316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.31.61.130:37317)
0812 16:39:27.480949 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:27.480956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:27.480986 (+ 30us) server_negotiation.cc:407] Connection header received
0812 16:39:27.481059 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:27.481066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:27.481319 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:27.481514 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:27.482808 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.483991 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.485268 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.485849 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.485948 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:27.488457 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:27.488466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:27.488480 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:27.488529 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:27.491049 (+ 2520us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.61.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 16:39:27.491354 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59690: Request ticket server kudu/127.31.61.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":326,"threads_started":1}
W20260812 16:39:27.491751 8967 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59690: Request ticket server kudu/127.31.61.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 16:39:27.491794 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8644: Not authorized: Client connection negotiation failed: client connection to 127.31.61.130:37317: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.61.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 16:39:27.491943 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 8644
I20260812 16:39:27.506912 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8780
I20260812 16:39:27.520258 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:27.508043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.31.61.131:42565)
0812 16:39:27.508604 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:27.508611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:27.508636 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:39:27.508703 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:27.508710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:27.508952 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:27.509142 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:27.510526 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.511567 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.512913 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:27.513699 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:27.513830 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:27.516665 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:27.516675 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:27.516690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:27.516735 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:27.519523 (+ 2788us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.61.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 16:39:27.519928 (+ 405us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48806: Request ticket server kudu/127.31.61.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
W20260812 16:39:27.520309 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8780: Not authorized: Client connection negotiation failed: client connection to 127.31.61.131:42565: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.61.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 16:39:27.520375 8968 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48806: Request ticket server kudu/127.31.61.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 16:39:27.520509 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 8780
I20260812 16:39:27.535696 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8415
I20260812 16:39:27.537221 8480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:27.657188 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 8415
2026-08-12T16:39:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21274 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T16:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:39:27Z Disabled control of system clock
I20260812 16:39:27.734333 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43277
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:43277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 16:39:28.084554 8973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:28.084844 8973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:28.084924 8973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:28.092336 8973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:39:28.092449 8973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:28.092486 8973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:28.092543 8973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:39:28.092586 8973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:39:28.102579 8973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:43277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:28.103914 8973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:28.105659 8973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:28.118096 8978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:28.118216 8981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:28.118124 8979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:28.118443 8973 server_base.cc:1034] running on GCE node
I20260812 16:39:28.119534 8973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:28.121026 8973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:28.122262 8973 hybrid_clock.cc:648] HybridClock initialized: now 1786552768122221 us; error 31 us; skew 500 ppm
I20260812 16:39:28.122645 8973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:39:28.125129 8973 webserver.cc:460] Webserver started at http://127.31.61.190:40769/ using document root <none> and password file <none>
I20260812 16:39:28.125833 8973 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:28.125941 8973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:28.126267 8973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:28.129452 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "0806b012be8c4cd28b8a916de56333d2"
format_stamp: "Formatted at 2026-08-12 16:39:28 on dist-test-slave-f7th"
server_key: "d05ce701d98c6186dbce410a0d9a63db"
server_key_iv: "4adee1cdda5678f82cf9e3cd82f0b8d1"
server_key_version: "encryptionkey@0"
I20260812 16:39:28.130426 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "0806b012be8c4cd28b8a916de56333d2"
format_stamp: "Formatted at 2026-08-12 16:39:28 on dist-test-slave-f7th"
server_key: "d05ce701d98c6186dbce410a0d9a63db"
server_key_iv: "4adee1cdda5678f82cf9e3cd82f0b8d1"
server_key_version: "encryptionkey@0"
I20260812 16:39:28.136260 8973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 16:39:28.140468 8987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:28.142117 8973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:39:28.142315 8973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "0806b012be8c4cd28b8a916de56333d2"
format_stamp: "Formatted at 2026-08-12 16:39:28 on dist-test-slave-f7th"
server_key: "d05ce701d98c6186dbce410a0d9a63db"
server_key_iv: "4adee1cdda5678f82cf9e3cd82f0b8d1"
server_key_version: "encryptionkey@0"
I20260812 16:39:28.142535 8973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:28.185387 8973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:28.186295 8973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:28.186589 8973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:28.204393 8973 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:43277
I20260812 16:39:28.204399 9039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:43277 every 8 connection(s)
I20260812 16:39:28.205952 8973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:39:28.209715 9040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:28.209806 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 8973
I20260812 16:39:28.209996 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:39:28.210704 31990 external_mini_cluster.cc:1428] Setting key fa76cd2bf3a64bacf1e46b2027b049f1
I20260812 16:39:28.223636 9040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2: Bootstrap starting.
I20260812 16:39:28.225485 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:28.212734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.31.61.190:43277)
0812 16:39:28.213681 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:28.213693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:28.213737 (+ 44us) server_negotiation.cc:407] Connection header received
0812 16:39:28.215946 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:28.215972 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:28.216490 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:28.216838 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:28.217547 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:28.218860 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:28.220785 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:28.221576 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:28.221725 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:28.223051 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:28.223060 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:28.223075 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:28.223704 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:28.223817 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:28.223848 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:28.223948 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:28.224151 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 16:39:28.224514 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":316,"threads_started":1}
I20260812 16:39:28.227432 9040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:28.228837 9040 log.cc:824] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:28.231937 9040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2: No bootstrap required, opened a new log
I20260812 16:39:28.239516 9040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0806b012be8c4cd28b8a916de56333d2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43277 } }
I20260812 16:39:28.239929 9040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:28.240026 9040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0806b012be8c4cd28b8a916de56333d2, State: Initialized, Role: FOLLOWER
I20260812 16:39:28.240578 9040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0806b012be8c4cd28b8a916de56333d2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43277 } }
I20260812 16:39:28.240811 9040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:39:28.240949 9040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:39:28.241117 9040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:28.243268 9040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0806b012be8c4cd28b8a916de56333d2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43277 } }
I20260812 16:39:28.243820 9040 leader_election.cc:302] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0806b012be8c4cd28b8a916de56333d2; no voters:
I20260812 16:39:28.244424 9040 leader_election.cc:288] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:39:28.244544 9045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:28.245532 9045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [term 1 LEADER]: Becoming Leader. State: Replica: 0806b012be8c4cd28b8a916de56333d2, State: Running, Role: LEADER
I20260812 16:39:28.246110 9045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0806b012be8c4cd28b8a916de56333d2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43277 } }
I20260812 16:39:28.246795 9040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:39:28.249045 9046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0806b012be8c4cd28b8a916de56333d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0806b012be8c4cd28b8a916de56333d2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43277 } } }
I20260812 16:39:28.249446 9046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:28.250291 9047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0806b012be8c4cd28b8a916de56333d2. Latest consensus state: current_term: 1 leader_uuid: "0806b012be8c4cd28b8a916de56333d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0806b012be8c4cd28b8a916de56333d2" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43277 } } }
I20260812 16:39:28.250651 9047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:28.252691 9054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:39:28.257699 9054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:39:28.266631 9054 catalog_manager.cc:1347] Generated new cluster ID: ed75bd6a80694f62b0ba40e2a9c4726b
I20260812 16:39:28.266744 9054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:39:28.277079 9054 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:39:28.278497 9054 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:39:28.299759 9054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0806b012be8c4cd28b8a916de56333d2: Generated new TSK 0
I20260812 16:39:28.300721 9054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:39:28.312230 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43277
--builtin_ntp_servers=127.31.61.148:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 16:39:28.660655 9065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:28.660941 9065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:28.661062 9065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:28.668320 9065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:28.668442 9065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:28.668588 9065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:39:28.677935 9065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:28.679267 9065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:28.681188 9065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:28.696172 9071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:28.696224 9070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:28.696246 9073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:28.696861 9065 server_base.cc:1034] running on GCE node
I20260812 16:39:28.697528 9065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:28.698513 9065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:28.699690 9065 hybrid_clock.cc:648] HybridClock initialized: now 1786552768699624 us; error 55 us; skew 500 ppm
I20260812 16:39:28.699990 9065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:39:28.708182 9065 webserver.cc:460] Webserver started at http://127.31.61.129:36779/ using document root <none> and password file <none>
I20260812 16:39:28.708937 9065 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:28.709046 9065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:28.709367 9065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:28.712297 9065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "29de678217b6479f8cd8f4d7f98dcbd9"
format_stamp: "Formatted at 2026-08-12 16:39:28 on dist-test-slave-f7th"
server_key: "5bdc64b25938b51ca6a4c5805f75c1fd"
server_key_iv: "1f6856e320598df8046ebc142027d0dd"
server_key_version: "encryptionkey@0"
I20260812 16:39:28.713106 9065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "29de678217b6479f8cd8f4d7f98dcbd9"
format_stamp: "Formatted at 2026-08-12 16:39:28 on dist-test-slave-f7th"
server_key: "5bdc64b25938b51ca6a4c5805f75c1fd"
server_key_iv: "1f6856e320598df8046ebc142027d0dd"
server_key_version: "encryptionkey@0"
I20260812 16:39:28.718636 9065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:39:28.722497 9079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:28.723984 9065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:39:28.724141 9065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "29de678217b6479f8cd8f4d7f98dcbd9"
format_stamp: "Formatted at 2026-08-12 16:39:28 on dist-test-slave-f7th"
server_key: "5bdc64b25938b51ca6a4c5805f75c1fd"
server_key_iv: "1f6856e320598df8046ebc142027d0dd"
server_key_version: "encryptionkey@0"
I20260812 16:39:28.724359 9065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:28.745730 9065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:28.746701 9065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:28.747015 9065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:28.748195 9065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:28.749837 9065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:28.749929 9065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:28.750034 9065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:28.750109 9065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:28.783272 9065 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:36013
I20260812 16:39:28.783353 9194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:36013 every 8 connection(s)
I20260812 16:39:28.784605 9065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:39:28.792289 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 9065
I20260812 16:39:28.792469 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:39:28.793088 31990 external_mini_cluster.cc:1428] Setting key 71f64e9873129f368c8eefaa755febd7
I20260812 16:39:28.795991 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43277
--builtin_ntp_servers=127.31.61.148:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 16:39:28.797696 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:28.787236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:44607 (local address 127.31.61.190:43277)
0812 16:39:28.787723 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:28.787729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:28.788892 (+ 1163us) server_negotiation.cc:407] Connection header received
0812 16:39:28.790641 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:28.790645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:28.790715 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:28.790814 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:28.792173 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:28.792993 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:28.794956 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:28.795342 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:28.795452 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:28.796828 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:28.796833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:28.796836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:28.796854 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:28.796903 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:28.796916 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:28.796998 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:28.797229 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 16:39:28.797343 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260812 16:39:28.798707 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:28.787860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43277 (local address 127.31.61.129:44607)
0812 16:39:28.788734 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:28.788776 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:28.790356 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:28.791105 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:28.791114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:28.791214 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:28.791949 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:28.791979 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:28.793166 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:28.793172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:28.794737 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:28.794756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:28.795576 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:28.795584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:28.795833 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:28.796514 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:39:28.796517 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:28.796558 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:39:28.796643 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:28.797011 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:28.797023 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:28.797032 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:39:28.797183 (+ 151us) client_negotiation.cc:240] Negotiation successful
0812 16:39:28.797604 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260812 16:39:28.800345 9195 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43277
I20260812 16:39:28.800660 9195 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:28.801451 9195 heartbeater.cc:511] Master 127.31.61.190:43277 requested a full tablet report, sending...
I20260812 16:39:28.803514 9004 ts_manager.cc:194] Registered new tserver with Master: 29de678217b6479f8cd8f4d7f98dcbd9 (127.31.61.129:36013)
I20260812 16:39:28.805267 9004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.129:44607
W20260812 16:39:29.130347 9199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:29.130678 9199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:29.130801 9199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:29.138789 9199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:29.138924 9199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:29.139060 9199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:39:29.149158 9199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:29.150509 9199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:29.152345 9199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:29.165511 9204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:29.165575 9205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:29.165632 9207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:29.166198 9199 server_base.cc:1034] running on GCE node
I20260812 16:39:29.166819 9199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:29.167806 9199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:29.169034 9199 hybrid_clock.cc:648] HybridClock initialized: now 1786552769168964 us; error 54 us; skew 500 ppm
I20260812 16:39:29.169358 9199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:39:29.171927 9199 webserver.cc:460] Webserver started at http://127.31.61.130:45173/ using document root <none> and password file <none>
I20260812 16:39:29.172640 9199 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:29.172730 9199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:29.173014 9199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:29.176023 9199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "75d1fb1399ba40bea74ab00604aac498"
format_stamp: "Formatted at 2026-08-12 16:39:29 on dist-test-slave-f7th"
server_key: "edb19da4c98eacf8dbea0ddf2c3e1157"
server_key_iv: "d3256e0b96fe897bc2cb5818da1dd57f"
server_key_version: "encryptionkey@0"
I20260812 16:39:29.176819 9199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "75d1fb1399ba40bea74ab00604aac498"
format_stamp: "Formatted at 2026-08-12 16:39:29 on dist-test-slave-f7th"
server_key: "edb19da4c98eacf8dbea0ddf2c3e1157"
server_key_iv: "d3256e0b96fe897bc2cb5818da1dd57f"
server_key_version: "encryptionkey@0"
I20260812 16:39:29.182294 9199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260812 16:39:29.185875 9213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:29.187242 9199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:39:29.187381 9199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "75d1fb1399ba40bea74ab00604aac498"
format_stamp: "Formatted at 2026-08-12 16:39:29 on dist-test-slave-f7th"
server_key: "edb19da4c98eacf8dbea0ddf2c3e1157"
server_key_iv: "d3256e0b96fe897bc2cb5818da1dd57f"
server_key_version: "encryptionkey@0"
I20260812 16:39:29.187578 9199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:29.210836 9199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:29.211916 9199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:29.212225 9199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:29.213238 9199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:29.214845 9199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:29.214944 9199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:29.215049 9199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:29.215132 9199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:29.248029 9199 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:32939
I20260812 16:39:29.248116 9329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:32939 every 8 connection(s)
I20260812 16:39:29.249347 9199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:39:29.251291 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 9199
I20260812 16:39:29.251480 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:39:29.252009 31990 external_mini_cluster.cc:1428] Setting key c79bb78ee3a486d2f1c027f506143b7d
I20260812 16:39:29.255093 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43277
--builtin_ntp_servers=127.31.61.148:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 16:39:29.261966 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:29.252028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:40555 (local address 127.31.61.190:43277)
0812 16:39:29.252233 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:29.252239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:29.253870 (+ 1631us) server_negotiation.cc:407] Connection header received
0812 16:39:29.255782 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:29.255788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:29.255868 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:29.255962 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:29.257017 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.257741 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.259264 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.259728 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.259846 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:29.261168 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:29.261173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:29.261176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:29.261194 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:29.261226 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:29.261237 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:29.261311 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:29.261543 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 16:39:29.261644 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:39:29.263170 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:29.252590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43277 (local address 127.31.61.130:40555)
0812 16:39:29.253706 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:29.253761 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:29.255498 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:29.256194 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:29.256203 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:29.256296 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:29.256853 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:29.256869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.257894 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.257898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:29.259122 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:29.259134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.259962 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.259971 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:29.260190 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:29.260839 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:39:29.260841 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:29.260881 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:39:29.260967 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:29.261325 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:29.261336 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:29.261345 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:39:29.261509 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 16:39:29.261907 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":305,"threads_started":1}
I20260812 16:39:29.264780 9330 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43277
I20260812 16:39:29.265074 9330 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:29.265822 9330 heartbeater.cc:511] Master 127.31.61.190:43277 requested a full tablet report, sending...
I20260812 16:39:29.267246 9004 ts_manager.cc:194] Registered new tserver with Master: 75d1fb1399ba40bea74ab00604aac498 (127.31.61.130:32939)
I20260812 16:39:29.268260 9004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.130:40555
W20260812 16:39:29.583774 9333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:29.584055 9333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:29.584160 9333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:29.591445 9333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:29.591553 9333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:29.591747 9333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:39:29.601408 9333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:29.602924 9333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:29.604842 9333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:29.617517 9338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:29.617728 9341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:29.617869 9339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:29.618929 9333 server_base.cc:1034] running on GCE node
I20260812 16:39:29.619592 9333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:29.620651 9333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:29.621873 9333 hybrid_clock.cc:648] HybridClock initialized: now 1786552769621775 us; error 84 us; skew 500 ppm
I20260812 16:39:29.622208 9333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:39:29.624703 9333 webserver.cc:460] Webserver started at http://127.31.61.131:35593/ using document root <none> and password file <none>
I20260812 16:39:29.625389 9333 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:29.625478 9333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:29.625771 9333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:29.628785 9333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "b5c749cb55d241fcb247c730f5aaf5b0"
format_stamp: "Formatted at 2026-08-12 16:39:29 on dist-test-slave-f7th"
server_key: "72222b21bb633448f10eb4b075c88206"
server_key_iv: "8a0688a8932193c32150d29c6f024435"
server_key_version: "encryptionkey@0"
I20260812 16:39:29.629653 9333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "b5c749cb55d241fcb247c730f5aaf5b0"
format_stamp: "Formatted at 2026-08-12 16:39:29 on dist-test-slave-f7th"
server_key: "72222b21bb633448f10eb4b075c88206"
server_key_iv: "8a0688a8932193c32150d29c6f024435"
server_key_version: "encryptionkey@0"
I20260812 16:39:29.635053 9333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 16:39:29.638612 9347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:29.640126 9333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:39:29.640290 9333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "b5c749cb55d241fcb247c730f5aaf5b0"
format_stamp: "Formatted at 2026-08-12 16:39:29 on dist-test-slave-f7th"
server_key: "72222b21bb633448f10eb4b075c88206"
server_key_iv: "8a0688a8932193c32150d29c6f024435"
server_key_version: "encryptionkey@0"
I20260812 16:39:29.640496 9333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:29.665773 9333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:29.666690 9333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:29.666985 9333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:29.667992 9333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:29.669487 9333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:29.669582 9333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:29.669698 9333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:29.669764 9333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:29.700296 9333 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:42931
I20260812 16:39:29.700372 9462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:42931 every 8 connection(s)
I20260812 16:39:29.701560 9333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:39:29.709293 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 9333
I20260812 16:39:29.709446 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:39:29.710018 31990 external_mini_cluster.cc:1428] Setting key 5808010b91491e62db249e9a5fe2a82c
I20260812 16:39:29.714133 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:29.704027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:46121 (local address 127.31.61.190:43277)
0812 16:39:29.704227 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:29.704233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:29.705729 (+ 1496us) server_negotiation.cc:407] Connection header received
0812 16:39:29.707474 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:29.707483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:29.707558 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:29.707692 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:29.708728 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.709526 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.711449 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.711931 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.712044 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:29.713336 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:29.713341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:29.713344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:29.713362 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:29.713393 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:29.713405 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:29.713475 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:29.713718 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 16:39:29.713816 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 16:39:29.715206 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:29.704568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43277 (local address 127.31.61.131:46121)
0812 16:39:29.705553 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:29.705592 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:29.707197 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:29.707885 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:29.707894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:29.707987 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:29.708556 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:29.708573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.709690 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.709696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:29.711317 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:29.711329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.712155 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.712162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:29.712386 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:29.713024 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:39:29.713026 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:29.713064 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:39:29.713144 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:29.713487 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:29.713499 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:29.713508 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:39:29.713660 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 16:39:29.714056 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":303,"threads_started":1}
I20260812 16:39:29.716607 9463 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43277
I20260812 16:39:29.716881 9463 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:29.717597 9463 heartbeater.cc:511] Master 127.31.61.190:43277 requested a full tablet report, sending...
I20260812 16:39:29.718979 9004 ts_manager.cc:194] Registered new tserver with Master: b5c749cb55d241fcb247c730f5aaf5b0 (127.31.61.131:42931)
I20260812 16:39:29.719841 9004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.131:46121
I20260812 16:39:29.725165 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 16:39:29.731112 9197 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:29.729626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.31.61.190:43277)
0812 16:39:29.729829 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:29.729836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:29.730224 (+ 388us) server_negotiation.cc:407] Connection header received
0812 16:39:29.730357 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:29.730361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:29.730435 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:29.730538 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:29.730881 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49840: BlockingRecv error: recv got EOF from 127.0.0.1:49840 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:39:29.731750 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 9065
I20260812 16:39:29.739497 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:29.732680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.31.61.129:36013)
0812 16:39:29.733120 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:29.733127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:29.733154 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:39:29.733210 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:29.733216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:29.733417 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:29.733549 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:29.733900 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.734882 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.736291 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.736893 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.736996 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:29.738017 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:29.738026 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:29.738040 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:29.738650 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:29.738708 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:29.738731 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:29.738822 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:29.739029 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 16:39:29.739170 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":212,"threads_started":1}
I20260812 16:39:29.740834 9190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:29.830807 9195 heartbeater.cc:503] Master 127.31.61.190:43277 was elected leader, sending a full tablet report...
I20260812 16:39:29.833675 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 9065
I20260812 16:39:29.853889 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 9199
I20260812 16:39:29.861591 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:29.854701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.31.61.130:32939)
0812 16:39:29.855246 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:29.855254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:29.855284 (+ 30us) server_negotiation.cc:407] Connection header received
0812 16:39:29.855347 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:29.855353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:29.855572 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:29.855730 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:29.856099 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.857101 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.858405 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.858982 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.859083 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:29.860147 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:29.860155 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:29.860168 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:29.860781 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:29.860828 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:29.860848 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:29.860954 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:29.861163 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 16:39:29.861295 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":280,"threads_started":1}
I20260812 16:39:29.862888 9325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:29.943575 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 9199
I20260812 16:39:29.957934 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 9333
I20260812 16:39:29.965890 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:29.958852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.31.61.131:42931)
0812 16:39:29.959458 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:29.959465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:29.959496 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:39:29.959559 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:29.959566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:29.959823 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:29.959961 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:29.960387 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.961472 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.962703 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:29.963297 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:29.963401 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:29.964517 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:29.964527 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:29.964543 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:29.965147 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:29.965196 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:29.965217 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:29.965308 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:29.965453 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 16:39:29.965587 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":282,"threads_started":1}
I20260812 16:39:29.967183 9457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:30.048768 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 9333
I20260812 16:39:30.063537 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 8973
I20260812 16:39:30.064731 9035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:30.168869 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 8973
2026-08-12T16:39:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2506 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T16:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:39:30Z Disabled control of system clock
I20260812 16:39:30.243782 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:35217
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 16:39:30.582916 9482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:30.583266 9482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:30.583389 9482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:30.591257 9482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:39:30.591429 9482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:30.591502 9482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:30.591589 9482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:39:30.591671 9482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:39:30.602075 9482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:35217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:30.603497 9482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:30.605522 9482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:30.617208 9488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:30.617259 9489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:30.617237 9491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:30.618049 9482 server_base.cc:1034] running on GCE node
I20260812 16:39:30.618903 9482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:30.620651 9482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:30.621978 9482 hybrid_clock.cc:648] HybridClock initialized: now 1786552770621927 us; error 56 us; skew 500 ppm
I20260812 16:39:30.622381 9482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:39:30.625186 9482 webserver.cc:460] Webserver started at http://127.31.61.190:33471/ using document root <none> and password file <none>
I20260812 16:39:30.625912 9482 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:30.626029 9482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:30.626362 9482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:30.629437 9482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "979d7156097b4ac8953fb7c25ac7c9f4"
format_stamp: "Formatted at 2026-08-12 16:39:30 on dist-test-slave-f7th"
server_key: "79b1192ec66d1bd277943fda88242850"
server_key_iv: "4139dd027f1b94b044291063a1677c7c"
server_key_version: "encryptionkey@0"
I20260812 16:39:30.630255 9482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "979d7156097b4ac8953fb7c25ac7c9f4"
format_stamp: "Formatted at 2026-08-12 16:39:30 on dist-test-slave-f7th"
server_key: "79b1192ec66d1bd277943fda88242850"
server_key_iv: "4139dd027f1b94b044291063a1677c7c"
server_key_version: "encryptionkey@0"
I20260812 16:39:30.635705 9482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 16:39:30.639822 9497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:30.641458 9482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 16:39:30.641660 9482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "979d7156097b4ac8953fb7c25ac7c9f4"
format_stamp: "Formatted at 2026-08-12 16:39:30 on dist-test-slave-f7th"
server_key: "79b1192ec66d1bd277943fda88242850"
server_key_iv: "4139dd027f1b94b044291063a1677c7c"
server_key_version: "encryptionkey@0"
I20260812 16:39:30.641969 9482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:30.648180 9482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:30.649017 9482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:30.649307 9482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:30.668239 9482 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:35217
I20260812 16:39:30.668287 9549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:35217 every 8 connection(s)
I20260812 16:39:30.669656 9482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:39:30.673990 9550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:30.677464 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 9482
I20260812 16:39:30.677734 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:39:30.678663 31990 external_mini_cluster.cc:1428] Setting key 539b3304ec4731f85dbe15f0a20e027a
I20260812 16:39:30.688019 9550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4: Bootstrap starting.
I20260812 16:39:30.688025 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:30.681137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.31.61.190:35217)
0812 16:39:30.682389 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:30.682400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:30.682447 (+ 47us) server_negotiation.cc:407] Connection header received
0812 16:39:30.684174 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:30.684201 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:30.684737 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:30.685188 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:30.686303 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:30.686309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:30.686322 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:30.686336 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:30.686393 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:30.686408 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:30.686628 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:30.686759 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 16:39:30.687022 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":389,"threads_started":1}
I20260812 16:39:30.691215 9550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:30.692288 9550 log.cc:824] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:30.695030 9550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4: No bootstrap required, opened a new log
I20260812 16:39:30.702463 9550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979d7156097b4ac8953fb7c25ac7c9f4" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35217 } }
I20260812 16:39:30.702845 9550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:30.702914 9550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979d7156097b4ac8953fb7c25ac7c9f4, State: Initialized, Role: FOLLOWER
I20260812 16:39:30.703488 9550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979d7156097b4ac8953fb7c25ac7c9f4" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35217 } }
I20260812 16:39:30.703763 9550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:39:30.703869 9550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:39:30.704001 9550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:30.706139 9550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979d7156097b4ac8953fb7c25ac7c9f4" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35217 } }
I20260812 16:39:30.706601 9550 leader_election.cc:302] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 979d7156097b4ac8953fb7c25ac7c9f4; no voters:
I20260812 16:39:30.707181 9550 leader_election.cc:288] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:39:30.707331 9555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:30.708417 9555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [term 1 LEADER]: Becoming Leader. State: Replica: 979d7156097b4ac8953fb7c25ac7c9f4, State: Running, Role: LEADER
I20260812 16:39:30.709096 9555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979d7156097b4ac8953fb7c25ac7c9f4" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35217 } }
I20260812 16:39:30.709425 9550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:39:30.712388 9556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "979d7156097b4ac8953fb7c25ac7c9f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979d7156097b4ac8953fb7c25ac7c9f4" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35217 } } }
I20260812 16:39:30.712596 9557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 979d7156097b4ac8953fb7c25ac7c9f4. Latest consensus state: current_term: 1 leader_uuid: "979d7156097b4ac8953fb7c25ac7c9f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979d7156097b4ac8953fb7c25ac7c9f4" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35217 } } }
I20260812 16:39:30.712826 9556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:30.712882 9557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:30.715011 9564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:39:30.720567 9564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:39:30.730741 9564 catalog_manager.cc:1347] Generated new cluster ID: f158471b045f429680b054a84486461f
I20260812 16:39:30.730881 9564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:39:30.744374 9564 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:39:30.744843 9564 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:39:30.754534 9564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 979d7156097b4ac8953fb7c25ac7c9f4: Generated new TSK 0
I20260812 16:39:30.755398 9564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:39:30.764297 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:35217
--builtin_ntp_servers=127.31.61.148:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 16:39:31.104156 9574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:31.104452 9574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:31.104570 9574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:31.112188 9574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:31.112317 9574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:31.112458 9574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:39:31.121979 9574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:35217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:31.123364 9574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:31.125242 9574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:31.139305 9580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:31.139263 9583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:31.139302 9581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:31.140290 9574 server_base.cc:1034] running on GCE node
I20260812 16:39:31.140894 9574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:31.141908 9574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:31.143146 9574 hybrid_clock.cc:648] HybridClock initialized: now 1786552771143090 us; error 46 us; skew 500 ppm
I20260812 16:39:31.143544 9574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:39:31.146036 9574 webserver.cc:460] Webserver started at http://127.31.61.129:33993/ using document root <none> and password file <none>
I20260812 16:39:31.146786 9574 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:31.146903 9574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:31.147220 9574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:31.153020 9574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "db5dce14e24546eb8d760b52d1d0051f"
format_stamp: "Formatted at 2026-08-12 16:39:31 on dist-test-slave-f7th"
server_key: "5763852a886c3bdffdf92a837c27df8a"
server_key_iv: "c6fa6b4a26e091a66faeb8f47a39103b"
server_key_version: "encryptionkey@0"
I20260812 16:39:31.153935 9574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "db5dce14e24546eb8d760b52d1d0051f"
format_stamp: "Formatted at 2026-08-12 16:39:31 on dist-test-slave-f7th"
server_key: "5763852a886c3bdffdf92a837c27df8a"
server_key_iv: "c6fa6b4a26e091a66faeb8f47a39103b"
server_key_version: "encryptionkey@0"
I20260812 16:39:31.159965 9574 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 16:39:31.164031 9590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:31.165566 9574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:39:31.165758 9574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "db5dce14e24546eb8d760b52d1d0051f"
format_stamp: "Formatted at 2026-08-12 16:39:31 on dist-test-slave-f7th"
server_key: "5763852a886c3bdffdf92a837c27df8a"
server_key_iv: "c6fa6b4a26e091a66faeb8f47a39103b"
server_key_version: "encryptionkey@0"
I20260812 16:39:31.166007 9574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:31.172528 9574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:31.173429 9574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:31.173717 9574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:31.174729 9574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:31.176280 9574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:31.176398 9574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:31.176522 9574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:31.176599 9574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:31.209723 9574 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:42467
I20260812 16:39:31.209754 9702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:42467 every 8 connection(s)
I20260812 16:39:31.211225 9574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:39:31.218614 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 9574
I20260812 16:39:31.218837 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:39:31.219504 31990 external_mini_cluster.cc:1428] Setting key 7d49af00a24611f5d7d300a9560df5a0
I20260812 16:39:31.219846 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:31.213931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:60259 (local address 127.31.61.190:35217)
0812 16:39:31.214422 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:31.214426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:31.215583 (+ 1157us) server_negotiation.cc:407] Connection header received
0812 16:39:31.217305 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:31.217310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:31.217376 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:31.217481 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:31.219025 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:31.219030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:31.219033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:31.219050 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:31.219093 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:31.219105 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:31.219213 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:31.219449 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 16:39:31.219519 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":248,"threads_started":1}
I20260812 16:39:31.220861 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:31.214474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:35217 (local address 127.31.61.129:60259)
0812 16:39:31.215413 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:31.215452 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:31.217040 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:31.217734 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:31.217742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:31.217833 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:31.218647 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:39:31.218650 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:31.218699 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:39:31.218813 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:31.219197 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:31.219207 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:31.219215 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:39:31.219480 (+ 265us) client_negotiation.cc:240] Negotiation successful
0812 16:39:31.219775 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":290,"threads_started":1}
I20260812 16:39:31.222438 9703 heartbeater.cc:347] Connected to a master server at 127.31.61.190:35217
I20260812 16:39:31.222784 9703 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:31.222815 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:35217
--builtin_ntp_servers=127.31.61.148:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 16:39:31.223518 9703 heartbeater.cc:511] Master 127.31.61.190:35217 requested a full tablet report, sending...
I20260812 16:39:31.225756 9514 ts_manager.cc:194] Registered new tserver with Master: db5dce14e24546eb8d760b52d1d0051f (127.31.61.129:42467)
W20260812 16:39:31.554951 9707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:31.555261 9707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:31.555383 9707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:31.562842 9707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:31.562969 9707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:31.563127 9707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:39:31.573138 9707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:35217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:31.574627 9707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:31.576607 9707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:31.594105 9714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:31.594014 9713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:31.594820 9716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:31.595367 9707 server_base.cc:1034] running on GCE node
I20260812 16:39:31.596136 9707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:31.597433 9707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:31.598745 9707 hybrid_clock.cc:648] HybridClock initialized: now 1786552771598680 us; error 74 us; skew 500 ppm
I20260812 16:39:31.599152 9707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:39:31.601468 9707 webserver.cc:460] Webserver started at http://127.31.61.130:35449/ using document root <none> and password file <none>
I20260812 16:39:31.602288 9707 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:31.602383 9707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:31.602677 9707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:31.605962 9707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "f3130ab660564eacaf3759a24468c8c9"
format_stamp: "Formatted at 2026-08-12 16:39:31 on dist-test-slave-f7th"
server_key: "5ac08083b3f995e12cf1417a6d0cecff"
server_key_iv: "8f9f2a25237365c6eb521a9561cf9af0"
server_key_version: "encryptionkey@0"
I20260812 16:39:31.606738 9707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "f3130ab660564eacaf3759a24468c8c9"
format_stamp: "Formatted at 2026-08-12 16:39:31 on dist-test-slave-f7th"
server_key: "5ac08083b3f995e12cf1417a6d0cecff"
server_key_iv: "8f9f2a25237365c6eb521a9561cf9af0"
server_key_version: "encryptionkey@0"
I20260812 16:39:31.612517 9707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:39:31.616204 9723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:31.617643 9707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 16:39:31.617813 9707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "f3130ab660564eacaf3759a24468c8c9"
format_stamp: "Formatted at 2026-08-12 16:39:31 on dist-test-slave-f7th"
server_key: "5ac08083b3f995e12cf1417a6d0cecff"
server_key_iv: "8f9f2a25237365c6eb521a9561cf9af0"
server_key_version: "encryptionkey@0"
I20260812 16:39:31.618034 9707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:31.624176 9707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:31.625037 9707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:31.625296 9707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:31.626276 9707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:31.627848 9707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:31.627944 9707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:31.628047 9707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:31.628120 9707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:31.659874 9707 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:39101
I20260812 16:39:31.659909 9837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:39101 every 8 connection(s)
I20260812 16:39:31.661283 9707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:39:31.668740 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 9707
I20260812 16:39:31.668927 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:39:31.669436 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:31.663847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:46611 (local address 127.31.61.190:35217)
0812 16:39:31.664087 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:31.664091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:31.665478 (+ 1387us) server_negotiation.cc:407] Connection header received
0812 16:39:31.667158 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:31.667163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:31.667217 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:31.667343 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:31.668656 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:31.668661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:31.668664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:31.668681 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:31.668713 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:31.668725 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:31.668833 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:31.669047 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 16:39:31.669117 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 16:39:31.669478 31990 external_mini_cluster.cc:1428] Setting key 70eaaaa999d3bfcb06db6b504726c6d5
I20260812 16:39:31.670656 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:31.664367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:35217 (local address 127.31.61.130:46611)
0812 16:39:31.665316 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:31.665352 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:31.666898 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:31.667524 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:31.667532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:31.667669 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:31.668337 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:39:31.668339 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:31.668378 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:39:31.668468 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:31.668815 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:31.668826 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:31.668834 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:39:31.669125 (+ 291us) client_negotiation.cc:240] Negotiation successful
0812 16:39:31.669387 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":301,"threads_started":1}
I20260812 16:39:31.672353 9838 heartbeater.cc:347] Connected to a master server at 127.31.61.190:35217
I20260812 16:39:31.672663 9838 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:31.673239 9838 heartbeater.cc:511] Master 127.31.61.190:35217 requested a full tablet report, sending...
I20260812 16:39:31.673463 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:35217
--builtin_ntp_servers=127.31.61.148:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 16:39:31.674656 9514 ts_manager.cc:194] Registered new tserver with Master: f3130ab660564eacaf3759a24468c8c9 (127.31.61.130:39101)
W20260812 16:39:32.007560 9841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:32.007906 9841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:32.008042 9841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:32.015736 9841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:32.015846 9841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:32.015995 9841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:39:32.025488 9841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:35217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:32.026804 9841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:32.028714 9841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:32.041576 9847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:32.041518 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:32.041610 9848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:32.042742 9841 server_base.cc:1034] running on GCE node
I20260812 16:39:32.043450 9841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:32.044476 9841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:32.045706 9841 hybrid_clock.cc:648] HybridClock initialized: now 1786552772045649 us; error 46 us; skew 500 ppm
I20260812 16:39:32.046152 9841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:39:32.048688 9841 webserver.cc:460] Webserver started at http://127.31.61.131:36111/ using document root <none> and password file <none>
I20260812 16:39:32.049407 9841 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:32.049513 9841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:32.049818 9841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:32.052917 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "de3dc8900be340ed87ff2b7e096f1f1e"
format_stamp: "Formatted at 2026-08-12 16:39:32 on dist-test-slave-f7th"
server_key: "7c9ec76382af56e3bbf1f8f434d30292"
server_key_iv: "5a62b597c1dde1814bf7b0958a58d3b6"
server_key_version: "encryptionkey@0"
I20260812 16:39:32.053728 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "de3dc8900be340ed87ff2b7e096f1f1e"
format_stamp: "Formatted at 2026-08-12 16:39:32 on dist-test-slave-f7th"
server_key: "7c9ec76382af56e3bbf1f8f434d30292"
server_key_iv: "5a62b597c1dde1814bf7b0958a58d3b6"
server_key_version: "encryptionkey@0"
I20260812 16:39:32.059057 9841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260812 16:39:32.062673 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:32.064092 9841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:39:32.064246 9841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "de3dc8900be340ed87ff2b7e096f1f1e"
format_stamp: "Formatted at 2026-08-12 16:39:32 on dist-test-slave-f7th"
server_key: "7c9ec76382af56e3bbf1f8f434d30292"
server_key_iv: "5a62b597c1dde1814bf7b0958a58d3b6"
server_key_version: "encryptionkey@0"
I20260812 16:39:32.064453 9841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:32.070031 9841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:32.070884 9841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:32.071158 9841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:32.072220 9841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:32.073707 9841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:32.073805 9841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:32.073921 9841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:32.073990 9841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:32.105520 9841 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:38483
I20260812 16:39:32.105571 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:38483 every 8 connection(s)
I20260812 16:39:32.106799 9841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:39:32.111048 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 9841
I20260812 16:39:32.111246 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:39:32.111868 31990 external_mini_cluster.cc:1428] Setting key 56b4ed49a8857cc991dbd2de1ef928b8
I20260812 16:39:32.115626 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:32.109328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:38531 (local address 127.31.61.190:35217)
0812 16:39:32.109556 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:32.109562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:32.111006 (+ 1444us) server_negotiation.cc:407] Connection header received
0812 16:39:32.113179 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:32.113184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:32.113244 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:32.113388 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:32.114823 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:32.114829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:32.114833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:32.114854 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:32.114896 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:32.114912 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:32.115057 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:32.115275 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 16:39:32.115330 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:39:32.116659 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:32.109804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:35217 (local address 127.31.61.131:38531)
0812 16:39:32.110809 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:32.110857 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:32.112881 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:32.113568 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:32.113577 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:32.113669 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:32.114448 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:39:32.114451 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:32.114494 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:39:32.114588 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:32.115064 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:32.115073 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:32.115080 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 16:39:32.115362 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 16:39:32.115701 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":284,"threads_started":1}
I20260812 16:39:32.118069 9970 heartbeater.cc:347] Connected to a master server at 127.31.61.190:35217
I20260812 16:39:32.118382 9970 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:32.118984 9970 heartbeater.cc:511] Master 127.31.61.190:35217 requested a full tablet report, sending...
I20260812 16:39:32.120484 9514 ts_manager.cc:194] Registered new tserver with Master: de3dc8900be340ed87ff2b7e096f1f1e (127.31.61.131:38483)
I20260812 16:39:32.128701 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 16:39:32.135905 9705 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:39:32.133951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.31.61.190:35217)
0812 16:39:32.134163 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:32.134171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:32.134843 (+ 672us) server_negotiation.cc:407] Connection header received
0812 16:39:32.135065 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:32.135070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:32.135129 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:32.135223 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:32.135593 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53546: BlockingRecv error: recv got EOF from 127.0.0.1:53546 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:39:32.136807 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 9574
I20260812 16:39:32.141167 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:32.137986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.31.61.129:42467)
0812 16:39:32.138505 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:32.138512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:32.138539 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:39:32.138600 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:32.138607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:32.138836 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:32.138962 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:32.140156 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:32.140167 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:32.140184 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:32.140210 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:32.140261 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:32.140278 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:32.140395 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:32.140647 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 16:39:32.140781 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
I20260812 16:39:32.142489 9698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:32.224432 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 9574
I20260812 16:39:32.240113 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 9707
I20260812 16:39:32.243774 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:32.241147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.31.61.130:39101)
0812 16:39:32.241648 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:32.241659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:32.241691 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:39:32.241768 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:32.241777 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:32.242074 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:32.242231 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:32.243119 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:32.243125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:32.243137 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:32.243150 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:32.243196 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:32.243213 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:32.243317 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:32.243368 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 16:39:32.243458 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":230,"threads_started":1}
I20260812 16:39:32.245059 9833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:32.330868 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 9707
I20260812 16:39:32.345367 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 9841
I20260812 16:39:32.349526 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:32.346450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.31.61.131:38483)
0812 16:39:32.347146 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:32.347153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:32.347185 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:39:32.347257 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:32.347263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:32.347484 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:32.347664 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:32.348773 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:32.348780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:32.348792 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:39:32.348805 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:32.348847 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:39:32.348863 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:32.349034 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:32.349124 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 16:39:32.349244 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":334,"threads_started":1}
I20260812 16:39:32.350872 9965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:32.433153 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 9841
I20260812 16:39:32.446926 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 9482
I20260812 16:39:32.448215 9545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:32.552623 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 9482
2026-08-12T16:39:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2383 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:39:32 dist-test-slave-f7th krb5kdc[9988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:39:32 dist-test-slave-f7th krb5kdc[9988](info): set up 2 sockets
Aug 12 16:39:32 dist-test-slave-f7th krb5kdc[9988](info): commencing operation
krb5kdc: starting...
W20260812 16:39:33.588901 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:39:33 dist-test-slave-f7th krb5kdc[9988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:39:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:39:33.848385 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:39851
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:39851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:34.187716 10001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:34.188017 10001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:34.188153 10001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:34.195562 10001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:39:34.195700 10001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:34.195779 10001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:34.195858 10001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:39:34.195921 10001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:39:34.205931 10001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:39851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:39851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:34.207312 10001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:34.208622 10001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:34.221191 10007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:34.221220 10009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:34.221247 10006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:34.222224 10001 server_base.cc:1034] running on GCE node
I20260812 16:39:34.223037 10001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:34.224545 10001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:34.225755 10001 hybrid_clock.cc:648] HybridClock initialized: now 1786552774225710 us; error 29 us; skew 500 ppm
I20260812 16:39:34.226102 10001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:34 dist-test-slave-f7th krb5kdc[9988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552774, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:34.234812 10001 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:39:34.236479 10001 webserver.cc:460] Webserver started at http://127.31.61.190:38865/ using document root <none> and password file <none>
I20260812 16:39:34.237186 10001 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:34.237308 10001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:34.237615 10001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:34.240700 10001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "a7df36f0341a4e4cad92f6f68d675148"
format_stamp: "Formatted at 2026-08-12 16:39:34 on dist-test-slave-f7th"
server_key: "2c0b2b2a870f253da2ad926d8aded964"
server_key_iv: "847d14812d3b6fe2a63a3674c52107ec"
server_key_version: "encryptionkey@0"
I20260812 16:39:34.241466 10001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "a7df36f0341a4e4cad92f6f68d675148"
format_stamp: "Formatted at 2026-08-12 16:39:34 on dist-test-slave-f7th"
server_key: "2c0b2b2a870f253da2ad926d8aded964"
server_key_iv: "847d14812d3b6fe2a63a3674c52107ec"
server_key_version: "encryptionkey@0"
I20260812 16:39:34.246985 10001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 16:39:34.251085 10016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:34.252696 10001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:39:34.252862 10001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "a7df36f0341a4e4cad92f6f68d675148"
format_stamp: "Formatted at 2026-08-12 16:39:34 on dist-test-slave-f7th"
server_key: "2c0b2b2a870f253da2ad926d8aded964"
server_key_iv: "847d14812d3b6fe2a63a3674c52107ec"
server_key_version: "encryptionkey@0"
I20260812 16:39:34.253067 10001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:34.287252 10001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:34.292025 10001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:34.292300 10001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:34.312528 10001 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:39851
I20260812 16:39:34.312579 10071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:39851 every 8 connection(s)
I20260812 16:39:34.314217 10001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:39:34.317916 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:34.323935 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 10001
I20260812 16:39:34.324189 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:39:34.324843 31990 external_mini_cluster.cc:1428] Setting key 06210100ad250f178887b847a0f4f34e
I20260812 16:39:34.329438 10072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148: Bootstrap starting.
I20260812 16:39:34.333245 10072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:34.334522 10072 log.cc:824] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:34.338209 10072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148: No bootstrap required, opened a new log
Aug 12 16:39:34 dist-test-slave-f7th krb5kdc[9988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:39:34.346196 10072 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7df36f0341a4e4cad92f6f68d675148" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39851 } }
I20260812 16:39:34.346573 10072 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:34.346675 10072 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7df36f0341a4e4cad92f6f68d675148, State: Initialized, Role: FOLLOWER
I20260812 16:39:34.347301 10072 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7df36f0341a4e4cad92f6f68d675148" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39851 } }
I20260812 16:39:34.347530 10072 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:39:34.347709 10072 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:39:34.347929 10072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:34.349052 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:34.327002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.31.61.190:39851)
0812 16:39:34.327800 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:34.327816 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:34.327868 (+ 52us) server_negotiation.cc:407] Connection header received
0812 16:39:34.330177 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:34.330208 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:34.330711 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:34.331142 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:34.332913 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:34.334828 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:34.336647 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:34.337433 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:34.337635 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:34.341596 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:34.341606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:34.341622 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:34.341671 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:34.344912 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:34.345623 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:34.345630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:34.345636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:34.345811 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:34.346169 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:34.346174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:34.346176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:34.346803 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:34.347162 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:34.347555 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 16:39:34.347990 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":320,"threads_started":1}
I20260812 16:39:34.350404 10072 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7df36f0341a4e4cad92f6f68d675148" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39851 } }
I20260812 16:39:34.351200 10072 leader_election.cc:302] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7df36f0341a4e4cad92f6f68d675148; no voters:
I20260812 16:39:34.352053 10072 leader_election.cc:288] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:39:34.352188 10077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:34.353353 10077 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [term 1 LEADER]: Becoming Leader. State: Replica: a7df36f0341a4e4cad92f6f68d675148, State: Running, Role: LEADER
I20260812 16:39:34.353942 10077 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7df36f0341a4e4cad92f6f68d675148" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39851 } }
I20260812 16:39:34.354426 10072 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:39:34.356695 10078 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7df36f0341a4e4cad92f6f68d675148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7df36f0341a4e4cad92f6f68d675148" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39851 } } }
I20260812 16:39:34.357209 10078 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:34.357115 10079 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7df36f0341a4e4cad92f6f68d675148. Latest consensus state: current_term: 1 leader_uuid: "a7df36f0341a4e4cad92f6f68d675148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7df36f0341a4e4cad92f6f68d675148" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39851 } } }
I20260812 16:39:34.357806 10079 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148 [sys.catalog]: This master's current role is: LEADER
I20260812 16:39:34.358652 10083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:39:34.364346 10083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:39:34.377972 10083 catalog_manager.cc:1347] Generated new cluster ID: dc1799cd235e43fa845bc32d15ebc3df
I20260812 16:39:34.378118 10083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:39:34.400911 10083 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:39:34.402335 10083 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:39:34.423164 10083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148: Generated new TSK 0
I20260812 16:39:34.424113 10083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:39:34.492235 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39851
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:34.835480 10098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:34.835882 10098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:34.836000 10098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:34.843590 10098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:34.843767 10098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:34.843888 10098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:39:34.854528 10098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:34.855979 10098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:34.857409 10098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:34.870990 10104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:34.871068 10103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:34.871050 10106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:34.872291 10098 server_base.cc:1034] running on GCE node
I20260812 16:39:34.872985 10098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:34.874035 10098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:34.875237 10098 hybrid_clock.cc:648] HybridClock initialized: now 1786552774875184 us; error 48 us; skew 500 ppm
I20260812 16:39:34.875627 10098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:34 dist-test-slave-f7th krb5kdc[9988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552774, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:34.879005 10098 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:39:34.880656 10098 webserver.cc:460] Webserver started at http://127.31.61.129:39105/ using document root <none> and password file <none>
I20260812 16:39:34.881383 10098 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:34.881513 10098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:34.881814 10098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:34.885305 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "339254b781c0468a95a3e770a9a186e6"
format_stamp: "Formatted at 2026-08-12 16:39:34 on dist-test-slave-f7th"
server_key: "1e14d7ba1889bf40a089cf15d5fe59c4"
server_key_iv: "7fbea8ad0089f0c0bc8cc8b45f26d96b"
server_key_version: "encryptionkey@0"
I20260812 16:39:34.886116 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "339254b781c0468a95a3e770a9a186e6"
format_stamp: "Formatted at 2026-08-12 16:39:34 on dist-test-slave-f7th"
server_key: "1e14d7ba1889bf40a089cf15d5fe59c4"
server_key_iv: "7fbea8ad0089f0c0bc8cc8b45f26d96b"
server_key_version: "encryptionkey@0"
I20260812 16:39:34.891743 10098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:39:34.895493 10113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:34.896975 10098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:39:34.897152 10098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "339254b781c0468a95a3e770a9a186e6"
format_stamp: "Formatted at 2026-08-12 16:39:34 on dist-test-slave-f7th"
server_key: "1e14d7ba1889bf40a089cf15d5fe59c4"
server_key_iv: "7fbea8ad0089f0c0bc8cc8b45f26d96b"
server_key_version: "encryptionkey@0"
I20260812 16:39:34.897343 10098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:34.910821 10098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:34.914886 10098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:34.915176 10098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:34.916180 10098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:34.917877 10098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:34.917958 10098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:34.918077 10098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:34.918133 10098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:34.949792 10098 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:39871
I20260812 16:39:34.949828 10230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:39871 every 8 connection(s)
I20260812 16:39:34.951064 10098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:39:34.959889 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 10098
I20260812 16:39:34.960095 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:39:34.960701 31990 external_mini_cluster.cc:1428] Setting key 343efd9032a3956a8aa3e53fffd473ee
Aug 12 16:39:34 dist-test-slave-f7th krb5kdc[9988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552774, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:39:34.972709 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:34.953652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:54487 (local address 127.31.61.190:39851)
0812 16:39:34.954125 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:34.954130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:34.955034 (+ 904us) server_negotiation.cc:407] Connection header received
0812 16:39:34.956922 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:34.956928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:34.957039 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:34.957151 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:34.958777 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:34.959555 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:34.961215 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:34.961760 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:34.961881 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:34.966034 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:34.966041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:34.966045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:34.966087 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:34.969104 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:34.969928 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:34.969934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:34.969937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:34.970084 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:34.970466 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:34.970470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:34.970473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:34.970807 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:34.971033 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:34.972115 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0812 16:39:34.972307 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":253,"threads_started":1}
I20260812 16:39:34.973887 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:34.954100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:39851 (local address 127.31.61.129:54487)
0812 16:39:34.954861 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:34.954900 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:34.956640 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:34.957377 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:34.957412 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:34.957978 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:34.958608 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:34.958626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:34.959718 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:34.959723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:34.961055 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:34.961068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:34.962002 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:34.962011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:34.962240 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:34.963031 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:34.963057 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:34.965771 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:34.969288 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:34.969297 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:34.969311 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:34.969798 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:34.970175 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:34.970178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:34.970181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:34.970368 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:34.971063 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:34.971077 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:34.971553 (+ 476us) client_negotiation.cc:769] Sending connection context
0812 16:39:34.971931 (+ 378us) client_negotiation.cc:240] Negotiation successful
0812 16:39:34.972542 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":332,"threads_started":1}
I20260812 16:39:34.975647 10231 heartbeater.cc:347] Connected to a master server at 127.31.61.190:39851
I20260812 16:39:34.975979 10231 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:34.976727 10231 heartbeater.cc:511] Master 127.31.61.190:39851 requested a full tablet report, sending...
I20260812 16:39:34.979113 10034 ts_manager.cc:194] Registered new tserver with Master: 339254b781c0468a95a3e770a9a186e6 (127.31.61.129:39871)
I20260812 16:39:34.981022 10034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:54487
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:39:35.021075 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39851
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:35.368575 10237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:35.368887 10237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:35.369006 10237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:35.376536 10237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:35.376672 10237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:35.376829 10237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:39:35.386752 10237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:35.388101 10237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:35.389408 10237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:35.402688 10245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:35.402720 10243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:35.402801 10242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:35.403718 10237 server_base.cc:1034] running on GCE node
I20260812 16:39:35.404392 10237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:35.405411 10237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:35.406620 10237 hybrid_clock.cc:648] HybridClock initialized: now 1786552775406558 us; error 51 us; skew 500 ppm
I20260812 16:39:35.406952 10237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:35 dist-test-slave-f7th krb5kdc[9988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552775, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:35.410303 10237 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:39:35.411865 10237 webserver.cc:460] Webserver started at http://127.31.61.130:42173/ using document root <none> and password file <none>
I20260812 16:39:35.412567 10237 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:35.412670 10237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:35.412971 10237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:35.416083 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "5f0b558e279d488ca35874fe436c7f6d"
format_stamp: "Formatted at 2026-08-12 16:39:35 on dist-test-slave-f7th"
server_key: "e699560ed75928c5d6dc3a1a84d8c356"
server_key_iv: "d503553629e29e5ca4a02d8249877378"
server_key_version: "encryptionkey@0"
I20260812 16:39:35.416903 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "5f0b558e279d488ca35874fe436c7f6d"
format_stamp: "Formatted at 2026-08-12 16:39:35 on dist-test-slave-f7th"
server_key: "e699560ed75928c5d6dc3a1a84d8c356"
server_key_iv: "d503553629e29e5ca4a02d8249877378"
server_key_version: "encryptionkey@0"
I20260812 16:39:35.422564 10237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:39:35.426379 10252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:35.427847 10237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:39:35.428016 10237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "5f0b558e279d488ca35874fe436c7f6d"
format_stamp: "Formatted at 2026-08-12 16:39:35 on dist-test-slave-f7th"
server_key: "e699560ed75928c5d6dc3a1a84d8c356"
server_key_iv: "d503553629e29e5ca4a02d8249877378"
server_key_version: "encryptionkey@0"
I20260812 16:39:35.428229 10237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:35.449937 10237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:35.454156 10237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:35.454468 10237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:35.455497 10237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:35.456956 10237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:35.457038 10237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:35.457157 10237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:35.457211 10237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:35.489048 10237 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:45921
I20260812 16:39:35.489100 10366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:45921 every 8 connection(s)
I20260812 16:39:35.490281 10237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:39:35.495360 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 10237
I20260812 16:39:35.495540 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:39:35.495967 31990 external_mini_cluster.cc:1428] Setting key ccb37c24fd7302effcf61030aef2e97c
Aug 12 16:39:35 dist-test-slave-f7th krb5kdc[9988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552775, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:39:35.510933 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:35.492881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:51173 (local address 127.31.61.190:39851)
0812 16:39:35.493319 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:35.493324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:35.494298 (+ 974us) server_negotiation.cc:407] Connection header received
0812 16:39:35.496217 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:35.496223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:35.496313 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:35.496412 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:35.498097 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:35.498832 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:35.500417 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:35.500849 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:35.500969 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:35.504469 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:35.504476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:35.504481 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:35.504524 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:35.507592 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:35.508420 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:35.508425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:35.508428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:35.508592 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:35.509016 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:35.509020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:35.509023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:35.509357 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:35.509509 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:35.510312 (+ 803us) server_negotiation.cc:299] Negotiation successful
0812 16:39:35.510569 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":212,"threads_started":1}
I20260812 16:39:35.511972 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:35.493419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:39851 (local address 127.31.61.130:51173)
0812 16:39:35.494147 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:35.494189 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:35.495927 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:35.496680 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:35.496693 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:35.497283 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:35.497915 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:35.497942 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:35.498991 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:35.498995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:35.500274 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:35.500286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:35.501089 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:35.501110 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:35.501325 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:35.502004 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:35.502029 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:35.504221 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:35.507781 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:35.507790 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:35.507804 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:35.508289 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:35.508702 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:35.508705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:35.508708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:35.508913 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:35.509506 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:35.509514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:35.509915 (+ 401us) client_negotiation.cc:769] Sending connection context
0812 16:39:35.510192 (+ 277us) client_negotiation.cc:240] Negotiation successful
0812 16:39:35.510668 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":268,"threads_started":1}
I20260812 16:39:35.513944 10367 heartbeater.cc:347] Connected to a master server at 127.31.61.190:39851
I20260812 16:39:35.514415 10367 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:35.515372 10367 heartbeater.cc:511] Master 127.31.61.190:39851 requested a full tablet report, sending...
I20260812 16:39:35.517135 10034 ts_manager.cc:194] Registered new tserver with Master: 5f0b558e279d488ca35874fe436c7f6d (127.31.61.130:45921)
I20260812 16:39:35.517966 10034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:51173
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:39:35.550849 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39851
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:39:35.896375 10373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:39:35.896673 10373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:39:35.896772 10373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:39:35.904243 10373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:39:35.904368 10373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:39:35.904484 10373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:39:35.914497 10373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:39851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:39:35.915813 10373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:39:35.917078 10373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:39:35.930305 10379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:35.930284 10378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:39:35.930501 10381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:39:35.931569 10373 server_base.cc:1034] running on GCE node
I20260812 16:39:35.932225 10373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:39:35.933393 10373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:39:35.934729 10373 hybrid_clock.cc:648] HybridClock initialized: now 1786552775934642 us; error 64 us; skew 500 ppm
I20260812 16:39:35.935204 10373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:39:35 dist-test-slave-f7th krb5kdc[9988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552775, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:39:35.938845 10373 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:39:35.940629 10373 webserver.cc:460] Webserver started at http://127.31.61.131:40515/ using document root <none> and password file <none>
I20260812 16:39:35.941385 10373 fs_manager.cc:359] Metadata directory not provided
I20260812 16:39:35.941497 10373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:39:35.941761 10373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:39:35.944852 10373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "bd50a9cff15d4a83a7a7c398f7840c87"
format_stamp: "Formatted at 2026-08-12 16:39:35 on dist-test-slave-f7th"
server_key: "059f25850585e700ac342c4418b00179"
server_key_iv: "eb16542691bb2f395076e3260e341e56"
server_key_version: "encryptionkey@0"
I20260812 16:39:35.945604 10373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "bd50a9cff15d4a83a7a7c398f7840c87"
format_stamp: "Formatted at 2026-08-12 16:39:35 on dist-test-slave-f7th"
server_key: "059f25850585e700ac342c4418b00179"
server_key_iv: "eb16542691bb2f395076e3260e341e56"
server_key_version: "encryptionkey@0"
I20260812 16:39:35.951176 10373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 16:39:35.956597 10388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:35.958820 10373 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:39:35.959009 10373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "bd50a9cff15d4a83a7a7c398f7840c87"
format_stamp: "Formatted at 2026-08-12 16:39:35 on dist-test-slave-f7th"
server_key: "059f25850585e700ac342c4418b00179"
server_key_iv: "eb16542691bb2f395076e3260e341e56"
server_key_version: "encryptionkey@0"
I20260812 16:39:35.959218 10373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:39:35.974887 10373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:39:35.979067 10373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:39:35.979324 10373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:39:35.980314 10373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:39:35.981725 10373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:39:35.981801 10373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:35.981894 10373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:39:35.981961 10373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:39:35.984545 10231 heartbeater.cc:503] Master 127.31.61.190:39851 was elected leader, sending a full tablet report...
I20260812 16:39:36.016539 10373 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:33657
I20260812 16:39:36.016577 10501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:33657 every 8 connection(s)
I20260812 16:39:36.017799 10373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:39:36.025529 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 10373
I20260812 16:39:36.025730 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:39:36.026181 31990 external_mini_cluster.cc:1428] Setting key 2fb50faf2fafcd2a861e066e329a2b53
Aug 12 16:39:36 dist-test-slave-f7th krb5kdc[9988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552775, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:39:36.041929 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.020628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:52605 (local address 127.31.61.190:39851)
0812 16:39:36.021156 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.021161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.022287 (+ 1126us) server_negotiation.cc:407] Connection header received
0812 16:39:36.024206 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.024211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.024297 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.024394 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:36.026208 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.027251 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.029397 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.029908 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.030021 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.034401 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:36.034406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:36.034410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:36.034447 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:36.037834 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:36.038865 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:36.038869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:36.038872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:36.039021 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:36.039553 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:36.039557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:36.039559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:36.039975 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:36.040133 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.041315 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.041556 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":232,"threads_started":1}
I20260812 16:39:36.043817 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.021355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:39851 (local address 127.31.61.131:52605)
0812 16:39:36.022114 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:36.022152 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:36.023856 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:36.024685 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:36.024695 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:36.025351 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:39:36.026016 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.026039 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.027411 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.027416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.029237 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.029252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.030311 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.030320 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.030732 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.031452 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:39:36.031477 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:39:36.034119 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:39:36.038024 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:36.038032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:36.038048 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:36.038710 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:36.039150 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:39:36.039153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:39:36.039156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:39:36.039378 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:39:36.040163 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:39:36.040172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:39:36.040630 (+ 458us) client_negotiation.cc:769] Sending connection context
0812 16:39:36.041213 (+ 583us) client_negotiation.cc:240] Negotiation successful
0812 16:39:36.042046 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":283,"threads_started":1}
I20260812 16:39:36.045554 10502 heartbeater.cc:347] Connected to a master server at 127.31.61.190:39851
I20260812 16:39:36.045899 10502 heartbeater.cc:464] Registering TS with master...
I20260812 16:39:36.047132 10502 heartbeater.cc:511] Master 127.31.61.190:39851 requested a full tablet report, sending...
I20260812 16:39:36.048758 10034 ts_manager.cc:194] Registered new tserver with Master: bd50a9cff15d4a83a7a7c398f7840c87 (127.31.61.131:33657)
I20260812 16:39:36.049621 10034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:52605
I20260812 16:39:36.055583 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:39:36.074051 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.061224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.31.61.190:39851)
0812 16:39:36.061383 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.061387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.061708 (+ 321us) server_negotiation.cc:407] Connection header received
0812 16:39:36.061867 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.061871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.061966 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.062043 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:36.063483 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.064295 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.065515 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.065923 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.066042 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.068296 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:36.068301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:36.068305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:36.068338 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:36.071164 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:36.071858 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:36.071864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:36.071867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:36.072016 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:36.072374 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:36.072380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:36.072384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:36.072867 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:36.073123 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.073493 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.073746 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 16:39:36.079573 10034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:39:36.087546 10034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:39:36.113360 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.103994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.31.61.130:45921)
0812 16:39:36.104631 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.104640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.105195 (+ 555us) server_negotiation.cc:407] Connection header received
0812 16:39:36.105309 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.105317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.105588 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.105770 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:36.106863 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.108206 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.110717 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.112214 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.112344 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.112602 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.112849 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.113018 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":308,"threads_started":1}
I20260812 16:39:36.113464 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.103997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:45921 (local address 127.0.0.1:43762)
0812 16:39:36.104718 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:36.104764 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:36.104987 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:36.106071 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:36.106079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:36.106110 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:36.106635 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.106652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.108330 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.108335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.110528 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.110547 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.112420 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.112426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.112623 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.112677 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 16:39:36.112790 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 16:39:36.113014 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":44,"thread_start_us":633,"threads_started":1}
I20260812 16:39:36.114847 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.104514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:33657 (local address 127.0.0.1:52912)
0812 16:39:36.105154 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:36.105171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:36.105283 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:36.106413 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:36.106418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:36.106450 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:36.106949 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.106961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.109471 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.109477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.111626 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.111642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.113813 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.113823 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.114019 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.114054 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:39:36.114140 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 16:39:36.114283 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":6,"thread_start_us":752,"threads_started":1}
I20260812 16:39:36.115814 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.105003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.31.61.131:33657)
0812 16:39:36.105840 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.105847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.105877 (+ 30us) server_negotiation.cc:407] Connection header received
0812 16:39:36.105982 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.105990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.106200 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.107259 (+ 1059us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:36.107555 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.109309 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.111865 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.113279 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.113438 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.113660 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.115286 (+ 1626us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.115469 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":515,"threads_started":1}
I20260812 16:39:36.118479 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.106187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.31.61.129:39871)
0812 16:39:36.109951 (+ 3764us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.109969 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.109999 (+ 30us) server_negotiation.cc:407] Connection header received
0812 16:39:36.110096 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.110104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.110398 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.110607 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:36.111455 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.112949 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.115726 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.117200 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.117428 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.117671 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.117980 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.118152 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3601,"thread_start_us":3431,"threads_started":1}
I20260812 16:39:36.118492 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.105615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:39871 (local address 127.0.0.1:37418)
0812 16:39:36.107460 (+ 1845us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:36.107479 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:36.107579 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:36.110877 (+ 3298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:36.110883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:36.110910 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:36.111265 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.111273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.113137 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.113147 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.115510 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.115526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.117584 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.117591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.117765 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.117798 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:39:36.117899 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 16:39:36.118055 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1720,"thread_start_us":1579,"threads_started":1}
I20260812 16:39:36.119236 10300 tablet_service.cc:1467] Processing CreateTablet for tablet 5d96d3200e2b4e259f167e6d4d007f02 (DEFAULT_TABLE table=test-table [id=85531786ad33460da84e322b3d2dfa09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:36.120985 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d96d3200e2b4e259f167e6d4d007f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:36.121284 10437 tablet_service.cc:1467] Processing CreateTablet for tablet 5d96d3200e2b4e259f167e6d4d007f02 (DEFAULT_TABLE table=test-table [id=85531786ad33460da84e322b3d2dfa09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:36.122864 10437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d96d3200e2b4e259f167e6d4d007f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:36.123001 10164 tablet_service.cc:1467] Processing CreateTablet for tablet 5d96d3200e2b4e259f167e6d4d007f02 (DEFAULT_TABLE table=test-table [id=85531786ad33460da84e322b3d2dfa09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:39:36.124527 10164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d96d3200e2b4e259f167e6d4d007f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:39:36.136188 10521 tablet_bootstrap.cc:492] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: Bootstrap starting.
I20260812 16:39:36.139716 10521 tablet_bootstrap.cc:654] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:36.140863 10521 log.cc:824] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:36.144646 10521 tablet_bootstrap.cc:492] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: No bootstrap required, opened a new log
I20260812 16:39:36.145107 10521 ts_tablet_manager.cc:1397] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.006s
I20260812 16:39:36.145344 10523 tablet_bootstrap.cc:492] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: Bootstrap starting.
I20260812 16:39:36.147235 10524 tablet_bootstrap.cc:492] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: Bootstrap starting.
I20260812 16:39:36.149223 10523 tablet_bootstrap.cc:654] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:36.150277 10524 tablet_bootstrap.cc:654] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: Neither blocks nor log segments found. Creating new log.
I20260812 16:39:36.150550 10523 log.cc:824] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:36.152040 10524 log.cc:824] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:39:36.155110 10523 tablet_bootstrap.cc:492] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: No bootstrap required, opened a new log
I20260812 16:39:36.155575 10523 ts_tablet_manager.cc:1397] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260812 16:39:36.156597 10524 tablet_bootstrap.cc:492] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: No bootstrap required, opened a new log
I20260812 16:39:36.156407 10521 raft_consensus.cc:348] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } }
I20260812 16:39:36.157015 10521 raft_consensus.cc:374] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:36.157163 10521 raft_consensus.cc:729] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0b558e279d488ca35874fe436c7f6d, State: Initialized, Role: FOLLOWER
I20260812 16:39:36.157186 10524 ts_tablet_manager.cc:1397] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260812 16:39:36.158224 10521 consensus_queue.cc:260] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } }
I20260812 16:39:36.165848 10367 heartbeater.cc:503] Master 127.31.61.190:39851 was elected leader, sending a full tablet report...
I20260812 16:39:36.165977 10524 raft_consensus.cc:348] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } }
I20260812 16:39:36.166447 10524 raft_consensus.cc:374] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:36.166541 10524 raft_consensus.cc:729] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339254b781c0468a95a3e770a9a186e6, State: Initialized, Role: FOLLOWER
I20260812 16:39:36.166944 10521 ts_tablet_manager.cc:1428] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: Time spent starting tablet: real 0.022s user 0.011s sys 0.008s
I20260812 16:39:36.167747 10524 consensus_queue.cc:260] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } }
I20260812 16:39:36.167879 10523 raft_consensus.cc:348] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } }
I20260812 16:39:36.168532 10523 raft_consensus.cc:374] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:39:36.168679 10523 raft_consensus.cc:729] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd50a9cff15d4a83a7a7c398f7840c87, State: Initialized, Role: FOLLOWER
I20260812 16:39:36.169469 10523 consensus_queue.cc:260] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } }
I20260812 16:39:36.170908 10502 heartbeater.cc:503] Master 127.31.61.190:39851 was elected leader, sending a full tablet report...
I20260812 16:39:36.171629 10523 ts_tablet_manager.cc:1428] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260812 16:39:36.174863 10524 ts_tablet_manager.cc:1428] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: Time spent starting tablet: real 0.018s user 0.012s sys 0.005s
W20260812 16:39:36.203395 10232 tablet.cc:2369] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:39:36.242250 10368 tablet.cc:2369] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:39:36.269521 10503 tablet.cc:2369] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:39:36.541672 10527 raft_consensus.cc:482] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:39:36.541973 10527 raft_consensus.cc:504] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } }
I20260812 16:39:36.543670 10527 leader_election.cc:288] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd50a9cff15d4a83a7a7c398f7840c87 (127.31.61.131:33657), 339254b781c0468a95a3e770a9a186e6 (127.31.61.129:39871)
I20260812 16:39:36.550961 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.544082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:52893 (local address 127.31.61.131:33657)
0812 16:39:36.544265 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.544270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.544674 (+ 404us) server_negotiation.cc:407] Connection header received
0812 16:39:36.544937 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.544944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.545030 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.545153 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:36.545964 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.546824 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.549107 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.549903 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.550141 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.550222 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.550572 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.550709 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 16:39:36.550989 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.544922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:38193 (local address 127.31.61.129:39871)
0812 16:39:36.545138 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.545144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.545472 (+ 328us) server_negotiation.cc:407] Connection header received
0812 16:39:36.545609 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.545614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.545698 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.545811 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:39:36.546440 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.547337 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.549238 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.550016 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.550179 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.550251 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.550622 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.550750 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 16:39:36.551188 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.543963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:33657 (local address 127.31.61.130:52893)
0812 16:39:36.544561 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:36.544578 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:36.544721 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:36.545248 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:36.545254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:36.545296 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:36.545784 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.545801 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.546989 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.546996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.548932 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.548945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.550196 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.550203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.550417 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.550442 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 16:39:36.550516 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:39:36.550655 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":21,"spinlock_wait_cycles":25344,"thread_start_us":306,"threads_started":1}
I20260812 16:39:36.551213 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.544736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:39871 (local address 127.31.61.130:38193)
0812 16:39:36.545366 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0812 16:39:36.545385 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:39:36.545498 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:39:36.545848 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:39:36.545852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:39:36.545879 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:39:36.546269 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.546280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.547550 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.547554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.549077 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:39:36.549092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.550259 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.550265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:39:36.550458 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.550481 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 16:39:36.550571 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 16:39:36.550701 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":33,"thread_start_us":316,"threads_started":1}
I20260812 16:39:36.552467 10457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d96d3200e2b4e259f167e6d4d007f02" candidate_uuid: "5f0b558e279d488ca35874fe436c7f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" is_pre_election: true
I20260812 16:39:36.552441 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d96d3200e2b4e259f167e6d4d007f02" candidate_uuid: "5f0b558e279d488ca35874fe436c7f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339254b781c0468a95a3e770a9a186e6" is_pre_election: true
I20260812 16:39:36.552991 10457 raft_consensus.cc:2454] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f0b558e279d488ca35874fe436c7f6d in term 0.
I20260812 16:39:36.552991 10184 raft_consensus.cc:2454] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f0b558e279d488ca35874fe436c7f6d in term 0.
I20260812 16:39:36.553691 10253 leader_election.cc:302] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f0b558e279d488ca35874fe436c7f6d, bd50a9cff15d4a83a7a7c398f7840c87; no voters:
I20260812 16:39:36.554090 10527 raft_consensus.cc:2781] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:39:36.554246 10527 raft_consensus.cc:482] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:39:36.554359 10527 raft_consensus.cc:3037] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:36.556599 10527 raft_consensus.cc:504] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } }
I20260812 16:39:36.557245 10527 leader_election.cc:288] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [CANDIDATE]: Term 1 election: Requested vote from peers bd50a9cff15d4a83a7a7c398f7840c87 (127.31.61.131:33657), 339254b781c0468a95a3e770a9a186e6 (127.31.61.129:39871)
I20260812 16:39:36.557763 10457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d96d3200e2b4e259f167e6d4d007f02" candidate_uuid: "5f0b558e279d488ca35874fe436c7f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd50a9cff15d4a83a7a7c398f7840c87"
I20260812 16:39:36.557793 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d96d3200e2b4e259f167e6d4d007f02" candidate_uuid: "5f0b558e279d488ca35874fe436c7f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339254b781c0468a95a3e770a9a186e6"
I20260812 16:39:36.558014 10457 raft_consensus.cc:3037] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:36.558013 10184 raft_consensus.cc:3037] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:39:36.560245 10184 raft_consensus.cc:2454] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f0b558e279d488ca35874fe436c7f6d in term 1.
I20260812 16:39:36.560287 10457 raft_consensus.cc:2454] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f0b558e279d488ca35874fe436c7f6d in term 1.
I20260812 16:39:36.560662 10253 leader_election.cc:302] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339254b781c0468a95a3e770a9a186e6, 5f0b558e279d488ca35874fe436c7f6d; no voters:
I20260812 16:39:36.560961 10527 raft_consensus.cc:2781] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:39:36.561403 10527 raft_consensus.cc:686] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 1 LEADER]: Becoming Leader. State: Replica: 5f0b558e279d488ca35874fe436c7f6d, State: Running, Role: LEADER
I20260812 16:39:36.561945 10527 consensus_queue.cc:237] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } }
I20260812 16:39:36.566697 10032 catalog_manager.cc:5495] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f0b558e279d488ca35874fe436c7f6d (127.31.61.130). New cstate: current_term: 1 leader_uuid: "5f0b558e279d488ca35874fe436c7f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b558e279d488ca35874fe436c7f6d" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 45921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 } health_report { overall_health: UNKNOWN } } }
I20260812 16:39:36.687356 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.680676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.31.61.130:45921)
0812 16:39:36.681377 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.681381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.681399 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:39:36.681631 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.681636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.681727 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.681998 (+ 271us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:39:36.682489 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.683419 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.685001 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.685414 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.685557 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.686100 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:39:36.686665 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:39:36.686783 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.686964 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.687043 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":497,"threads_started":1}
I20260812 16:39:36.695986 10184 raft_consensus.cc:1261] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [term 1 FOLLOWER]: Refusing update from remote peer 5f0b558e279d488ca35874fe436c7f6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:36.696063 10457 raft_consensus.cc:1261] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [term 1 FOLLOWER]: Refusing update from remote peer 5f0b558e279d488ca35874fe436c7f6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:39:36.696959 10532 consensus_queue.cc:1035] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "339254b781c0468a95a3e770a9a186e6" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:36.697321 10527 consensus_queue.cc:1035] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd50a9cff15d4a83a7a7c398f7840c87" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 33657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:39:36.711619 10537 mvcc.cc:205] Tried to move back new op lower bound from 7317720173335617536 to 7317720172802678784. Current Snapshot: MvccSnapshot[applied={T|T < 7317720173335617536}]
I20260812 16:39:36.714162 10540 mvcc.cc:205] Tried to move back new op lower bound from 7317720173335617536 to 7317720172802678784. Current Snapshot: MvccSnapshot[applied={T|T < 7317720173335617536}]
I20260812 16:39:36.716568 10541 mvcc.cc:205] Tried to move back new op lower bound from 7317720173335617536 to 7317720172802678784. Current Snapshot: MvccSnapshot[applied={T|T < 7317720173335617536 or (T in {7317720173335617536})}]
I20260812 16:39:36.724681 10033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:36.724926 10033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:39:36.728513 10033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7df36f0341a4e4cad92f6f68d675148: Sending DeleteTablet for 3 replicas of tablet 5d96d3200e2b4e259f167e6d4d007f02
I20260812 16:39:36.729640 10300 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d96d3200e2b4e259f167e6d4d007f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:36 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:43762
I20260812 16:39:36.729815 10164 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d96d3200e2b4e259f167e6d4d007f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:36 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:37418
I20260812 16:39:36.729851 10437 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d96d3200e2b4e259f167e6d4d007f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:39:36 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:52912
I20260812 16:39:36.730580 10548 tablet_replica.cc:330] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: stopping tablet replica
I20260812 16:39:36.731514 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 10098
I20260812 16:39:36.731979 10549 tablet_replica.cc:330] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: stopping tablet replica
I20260812 16:39:36.732388 10549 raft_consensus.cc:2229] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:36.732723 10550 tablet_replica.cc:330] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: stopping tablet replica
I20260812 16:39:36.733026 10550 raft_consensus.cc:2229] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:39:36.733494 10550 raft_consensus.cc:2258] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:36.733423 10548 raft_consensus.cc:2229] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:39:36.734099 10548 raft_consensus.cc:2258] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:36.734169 10549 raft_consensus.cc:2258] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:39:36.735472 10550 ts_tablet_manager.cc:1905] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:36.735846 10549 ts_tablet_manager.cc:1905] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:36.736001 10548 ts_tablet_manager.cc:1905] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:39:36.742902 10550 ts_tablet_manager.cc:1918] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:39:36.742908 10549 ts_tablet_manager.cc:1918] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:39:36.743088 10549 log.cc:1196] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/5d96d3200e2b4e259f167e6d4d007f02
I20260812 16:39:36.743227 10550 log.cc:1196] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/5d96d3200e2b4e259f167e6d4d007f02
I20260812 16:39:36.743688 10549 ts_tablet_manager.cc:1939] T 5d96d3200e2b4e259f167e6d4d007f02 P 339254b781c0468a95a3e770a9a186e6: Deleting consensus metadata
I20260812 16:39:36.743847 10550 ts_tablet_manager.cc:1939] T 5d96d3200e2b4e259f167e6d4d007f02 P bd50a9cff15d4a83a7a7c398f7840c87: Deleting consensus metadata
I20260812 16:39:36.743983 10548 ts_tablet_manager.cc:1918] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:39:36.744202 10548 log.cc:1196] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/5d96d3200e2b4e259f167e6d4d007f02
I20260812 16:39:36.744782 10548 ts_tablet_manager.cc:1939] T 5d96d3200e2b4e259f167e6d4d007f02 P 5f0b558e279d488ca35874fe436c7f6d: Deleting consensus metadata
I20260812 16:39:36.745707 10017 catalog_manager.cc:4841] TS bd50a9cff15d4a83a7a7c398f7840c87 (127.31.61.131:33657): tablet 5d96d3200e2b4e259f167e6d4d007f02 (table test-table [id=85531786ad33460da84e322b3d2dfa09]) successfully deleted
I20260812 16:39:36.746214 10017 catalog_manager.cc:4841] TS 339254b781c0468a95a3e770a9a186e6 (127.31.61.129:39871): tablet 5d96d3200e2b4e259f167e6d4d007f02 (table test-table [id=85531786ad33460da84e322b3d2dfa09]) successfully deleted
I20260812 16:39:36.746570 10020 catalog_manager.cc:4841] TS 5f0b558e279d488ca35874fe436c7f6d (127.31.61.130:45921): tablet 5d96d3200e2b4e259f167e6d4d007f02 (table test-table [id=85531786ad33460da84e322b3d2dfa09]) successfully deleted
Aug 12 16:39:36 dist-test-slave-f7th krb5kdc[9988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:39:36.762809 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.732404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.31.61.129:39871)
0812 16:39:36.732582 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.732588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.748217 (+ 15629us) server_negotiation.cc:407] Connection header received
0812 16:39:36.748362 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.748368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.748455 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.748577 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:36.749787 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.750989 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.752483 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.752887 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.753006 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.757049 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:36.757066 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:36.757082 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:36.757126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:36.759956 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:36.760662 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:36.760670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:36.760672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:36.760816 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:36.761219 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:36.761223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:36.761225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:36.761598 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:36.761910 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.762228 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.762449 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:39:36.763716 10226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:36.855587 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 10098
I20260812 16:39:36.871848 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 10237
Aug 12 16:39:36 dist-test-slave-f7th krb5kdc[9988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:39:36.887693 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:36.872738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.31.61.130:45921)
0812 16:39:36.872915 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:36.872921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:36.872941 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:39:36.873135 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:36.873140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:36.873210 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:36.873302 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:36.874579 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.875344 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.876586 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:36.877040 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:36.877164 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:36.881238 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:36.881253 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:36.881270 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:36.881307 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:36.884660 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:36.885379 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:36.885386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:36.885389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:36.885627 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:36.886051 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:36.886056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:36.886060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:36.886426 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:36.886769 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:36.887036 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 16:39:36.887280 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 16:39:36.888399 10362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:37.006166 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 10237
I20260812 16:39:37.022919 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 10373
Aug 12 16:39:37 dist-test-slave-f7th krb5kdc[9988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:39:37.038075 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:39:37.023672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.31.61.131:33657)
0812 16:39:37.023845 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 16:39:37.023850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:39:37.023895 (+ 45us) server_negotiation.cc:407] Connection header received
0812 16:39:37.024118 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:39:37.024122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:39:37.024197 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:39:37.024280 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:39:37.025527 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:37.026343 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:37.027579 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:39:37.028145 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:39:37.028253 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:39:37.032659 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:39:37.032671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:39:37.032687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:39:37.032723 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:39:37.035455 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:37.036153 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:37.036160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:37.036162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:37.036294 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:39:37.036676 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:39:37.036680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:39:37.036682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:39:37.037051 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:39:37.037330 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 16:39:37.037643 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 16:39:37.037811 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 16:39:37.038774 10497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:37.128912 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 10373
I20260812 16:39:37.144039 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 10001
I20260812 16:39:37.145547 10067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:39:37.258912 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 10001
2026-08-12T16:39:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4709 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:39:37 dist-test-slave-f7th krb5kdc[10558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:39:37 dist-test-slave-f7th krb5kdc[10558](info): set up 2 sockets
Aug 12 16:39:37 dist-test-slave-f7th krb5kdc[10558](info): commencing operation
krb5kdc: starting...
W20260812 16:39:38.310526 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:39:38 dist-test-slave-f7th krb5kdc[10558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:39:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.61.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.61.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/rangeradmin_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/rangeradmin_127.31.61.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.61.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.61.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/rangerlookup_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/rangerlookup_127.31.61.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.61.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.61.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/HTTP_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/HTTP_127.31.61.148@KRBTEST.COM.keytab.
I20260812 16:39:38.673174 31990 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgT9LAJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/postgres -l logfile start
2026-08-12 16:39:41.606 UTC [10590] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 16:39:41.606 UTC [10590] LOG: listening on IPv4 address "127.31.61.148", port 59707
2026-08-12 16:39:41.611 UTC [10590] LOG: listening on Unix socket "/tmp/.s.PGSQL.59707"
2026-08-12 16:39:41.634 UTC [10593] LOG: database system was shut down at 2026-08-12 16:39:39 UTC
2026-08-12 16:39:41.638 UTC [10590] LOG: database system is ready to accept connections
I20260812 16:39:42.551802 31990 mini_postgres.cc:96] Postgres bound to 59707
2026-08-12 16:39:42.571 UTC [10601] FATAL: database "slave" does not exist
127.31.61.148:59707 - accepting connections
I20260812 16:39:42.572532 31990 mini_ranger.cc:161] Starting Ranger...
I20260812 16:39:42.604450 31990 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 16:39:43.270605 31990 mini_ranger.cc:87] Created ranger Postgres database
I20260812 16:39:43.270716 31990 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-admin
2026-08-12 16:39:43,661 [I] DB FLAVOR :POSTGRES
2026-08-12 16:39:43,662 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 16:39:43,662 [I] Checking connection..
2026-08-12 16:39:43,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 16:39:43,925 [I] Checking connection passed.
2026-08-12 16:39:43,926 [I] --------- Verifying version history table ---------
2026-08-12 16:39:43,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:39:44,177 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 16:39:44,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:39:44,419 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 16:39:44,420 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 16:39:44,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 16:39:44.674 UTC [10665] WARNING: there is no transaction in progress
2026-08-12 16:39:44,680 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 16:39:44,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:39:44,953 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 16:39:44,953 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 16:39:44,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 16:39:45,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 16:39:45,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f7th.c.gcp-upstream.internal','N') ;"
2026-08-12 16:39:45,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 16:39:45,999 [I] Table x_portal_user does not exist in database ranger
2026-08-12 16:39:45,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 16:39:46,261 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 16:39:46,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 16:39:46,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 16:39:46,757 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 16:39:46,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 16:39:47.838 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:47.858 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:47.880 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:47.905 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:47.926 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:47.945 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.009 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.017 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.033 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.045 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.056 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.072 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.082 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.089 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.096 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48.105 UTC [10774] WARNING: there is no transaction in progress
2026-08-12 16:39:48,899 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 16:39:48,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 16:39:49,132 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 16:39:49,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 16:39:49,402 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 16:39:49,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 16:39:49,635 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 16:39:49,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 16:39:49,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 16:39:50,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f7th.c.gcp-upstream.internal';"
2026-08-12 16:39:50,404 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 16:39:50,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 16:39:50,650 [I] DB_PATCHES have already been applied
I20260812 16:39:50.656409 31990 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgT9LAJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 16:39:50.656502 31990 mini_ranger.cc:193] Using host: 127.31.61.148
I20260812 16:39:50.672700 31990 mini_ranger.cc:227] Ranger admin URL: http://127.31.61.148:44645
Aug 12, 2026 4:39:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 4:39:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 4:39:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 4:39:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 4:39:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 4:39:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 4:39:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 16:39:52.833531 31990 mini_ranger.cc:160] Time spent starting Ranger: real 10.261s user 0.001s sys 0.059s
Aug 12, 2026 4:40:13 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 4:40:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 4:40:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 4:40:13 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 4:40:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 16:40:15.090668 31990 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.61.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.61.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/rangerkms_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/rangerkms_127.31.61.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.61.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.61.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.61.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/HTTP_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.61.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/HTTP_127.31.61.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 16:40:15 dist-test-slave-f7th krb5kdc[10558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552815, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 16:40:15.293362 31990 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 16:40:15.318686 31990 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 16:40:16.537237 31990 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 16:40:16.537396 31990 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-kms
I20260812 16:40:16.537544 31990 mini_ranger_kms.cc:227] Using postgres at 127.31.61.148:59707
2026-08-12 16:40:17,352 [I] DB FLAVOR :POSTGRES
2026-08-12 16:40:17,353 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 16:40:17,353 [I] Checking connection
2026-08-12 16:40:17,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 16:40:17,617 [I] connection success
2026-08-12 16:40:17,617 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 16:40:17,617 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 16:40:17,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 16:40:17,854 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 16:40:17,854 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 16:40:17,854 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 16:40:17,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:59707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 16:40:18,124 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 16:40:18.635372 31990 mini_ranger_kms.cc:310] Created kms service
I20260812 16:40:18.814381 31990 mini_ranger_kms.cc:326] Created kudu user
I20260812 16:40:18.867465 31990 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 16:40:18 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552815, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:40:19.281611 31990 mini_ranger_kms.cc:384] Added ranger policy
I20260812 16:40:19.283383 31990 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 16:40:19.283488 31990 mini_ranger_kms.cc:240] Using host: 127.31.61.148
I20260812 16:40:19.303455 31990 mini_ranger_kms.cc:276] Ranger KMS PID: 10958
I20260812 16:40:19.303735 31990 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.61.148:52697
Aug 12, 2026 4:40:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 4:40:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 4:40:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 4:40:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 4:40:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 4:40:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260812 16:40:21.464494 31990 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.171s user 0.130s sys 0.454s
I20260812 16:40:21.464831 31990 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 16:40:21.465056 31990 mini_ranger_kms.cc:401] 127.31.61.148:52697/kms/v1/keys
Aug 12 16:40:21 dist-test-slave-f7th krb5kdc[10558](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786552821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.61.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 16:40:22 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.61.148@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
Aug 12 16:40:22 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.61.148@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
Aug 12 16:40:24 dist-test-slave-f7th krb5kdc[10558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:40:24.779068 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43723
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:43723
--ranger_config_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:52697/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:40:25.168720 11010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:40:25.169019 11010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:40:25.169111 11010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:40:25.177480 11010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:40:25.177619 11010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:40:25.177666 11010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:40:25.177727 11010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:40:25.177778 11010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:40:25.189011 11010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.61.148:52697/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:43723
--ranger_config_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:40:25.190289 11010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:40:25.191617 11010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:40:25.204588 11015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:40:25.204545 11016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:40:25.204617 11018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:40:25.206075 11010 server_base.cc:1034] running on GCE node
I20260812 16:40:25.206997 11010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:40:25.208606 11010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:40:25.209846 11010 hybrid_clock.cc:648] HybridClock initialized: now 1786552825209798 us; error 40 us; skew 500 ppm
I20260812 16:40:25.210210 11010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:40:25 dist-test-slave-f7th krb5kdc[10558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:40:25.217657 11010 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:40:25.219462 11010 webserver.cc:460] Webserver started at http://127.31.61.190:32889/ using document root <none> and password file <none>
I20260812 16:40:25.220206 11010 fs_manager.cc:359] Metadata directory not provided
I20260812 16:40:25.220323 11010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:40:25.220626 11010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:40:25 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:40:25.282467 11010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "93f51e0ae45942e0a76225ed8e769637"
format_stamp: "Formatted at 2026-08-12 16:40:25 on dist-test-slave-f7th"
server_key: "063be2670f915e4ac788b739b0ce289f"
server_key_iv: "d8b3338849a32e2afee15cd12550bb86"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:25.283376 11010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "93f51e0ae45942e0a76225ed8e769637"
format_stamp: "Formatted at 2026-08-12 16:40:25 on dist-test-slave-f7th"
server_key: "063be2670f915e4ac788b739b0ce289f"
server_key_iv: "d8b3338849a32e2afee15cd12550bb86"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:25.288936 11010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260812 16:40:25.309617 11025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:40:25.311096 11010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 16:40:25.311271 11010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "93f51e0ae45942e0a76225ed8e769637"
format_stamp: "Formatted at 2026-08-12 16:40:25 on dist-test-slave-f7th"
server_key: "063be2670f915e4ac788b739b0ce289f"
server_key_iv: "d8b3338849a32e2afee15cd12550bb86"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:25.311489 11010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:40:25.348196 11010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:40:25.352465 11010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:40:25.352763 11010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:40:25.378793 11010 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:43723
I20260812 16:40:25.379139 11077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:43723 every 8 connection(s)
I20260812 16:40:25.380476 11010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:40:25.386574 11078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:40:25.389748 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 11010
I20260812 16:40:25.389974 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:40:25.390740 31990 external_mini_cluster.cc:1428] Setting key 2c11c84d25bb7460eda29d139ae402b5
I20260812 16:40:25.403249 11078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637: Bootstrap starting.
I20260812 16:40:25.408193 11078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637: Neither blocks nor log segments found. Creating new log.
I20260812 16:40:25.409914 11078 log.cc:824] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637: Log is configured to *not* fsync() on all Append() calls
I20260812 16:40:25.414219 11078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637: No bootstrap required, opened a new log
I20260812 16:40:25.423434 11078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f51e0ae45942e0a76225ed8e769637" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43723 } }
I20260812 16:40:25.424149 11078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:40:25.424263 11078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f51e0ae45942e0a76225ed8e769637, State: Initialized, Role: FOLLOWER
Aug 12 16:40:25 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:40:25.424924 11078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f51e0ae45942e0a76225ed8e769637" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43723 } }
I20260812 16:40:25.425396 11078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:40:25.425870 11078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:40:25.426262 11078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:40:25.430496 11078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f51e0ae45942e0a76225ed8e769637" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43723 } }
I20260812 16:40:25.431143 11078 leader_election.cc:302] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f51e0ae45942e0a76225ed8e769637; no voters:
I20260812 16:40:25.433125 11078 leader_election.cc:288] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:40:25.435359 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:25.393236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.31.61.190:43723)
0812 16:40:25.408635 (+ 15399us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:25.408653 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:25.408711 (+ 58us) server_negotiation.cc:407] Connection header received
0812 16:40:25.412015 (+ 3304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:25.412071 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:25.412613 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:25.413114 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:40:25.414707 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:25.416945 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:25.419082 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:25.420219 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:25.420409 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:25.425739 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:40:25.425754 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:40:25.425777 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:40:25.425840 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:40:25.430042 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:25.430978 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:25.430987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:25.430998 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:25.431214 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:25.431724 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:25.431730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:25.431734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:25.432547 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:40:25.432862 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:25.433366 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 16:40:25.433852 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15128,"thread_start_us":14878,"threads_started":1}
I20260812 16:40:25.436391 11084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:40:25.438144 11078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:40:25.439996 11084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [term 1 LEADER]: Becoming Leader. State: Replica: 93f51e0ae45942e0a76225ed8e769637, State: Running, Role: LEADER
I20260812 16:40:25.440728 11084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f51e0ae45942e0a76225ed8e769637" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43723 } }
I20260812 16:40:25.443655 11085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f51e0ae45942e0a76225ed8e769637. Latest consensus state: current_term: 1 leader_uuid: "93f51e0ae45942e0a76225ed8e769637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f51e0ae45942e0a76225ed8e769637" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43723 } } }
I20260812 16:40:25.444095 11085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [sys.catalog]: This master's current role is: LEADER
I20260812 16:40:25.443975 11083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93f51e0ae45942e0a76225ed8e769637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f51e0ae45942e0a76225ed8e769637" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43723 } } }
I20260812 16:40:25.444329 11083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637 [sys.catalog]: This master's current role is: LEADER
I20260812 16:40:25.460248 11090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:40:25.474653 11090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:40:25.476145 11078 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 16:40:25.494925 11090 catalog_manager.cc:1347] Generated new cluster ID: 7c270761539240728ebd08ee867478f9
I20260812 16:40:25.495203 11090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:40:25.522673 11090 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:40:25.525305 11090 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:40:25.539073 11090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637: Generated new TSK 0
I20260812 16:40:25.540421 11090 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:40:26.377898 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:52697/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43723
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:40:26 dist-test-slave-f7th krb5kdc[10558](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786552826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 16:40:26.924778 11124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:40:26.925383 11124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:40:26.925616 11124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:40:26.936492 11124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:40:26.936640 11124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:40:26.936794 11124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:40:26.948210 11124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.61.148:52697/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:40:26.949555 11124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:40:26.950937 11124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:40:26.966786 11135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:40:26.966998 11134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:40:26.967525 11137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:40:26.980062 11124 server_base.cc:1034] running on GCE node
I20260812 16:40:26.981215 11124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:40:26.982782 11124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:40:26.984206 11124 hybrid_clock.cc:648] HybridClock initialized: now 1786552826984079 us; error 101 us; skew 500 ppm
I20260812 16:40:26.984786 11124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:40:26 dist-test-slave-f7th krb5kdc[10558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:40:26.990518 11124 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:40:26.993444 11124 webserver.cc:460] Webserver started at http://127.31.61.129:35575/ using document root <none> and password file <none>
I20260812 16:40:26.994701 11124 fs_manager.cc:359] Metadata directory not provided
I20260812 16:40:26.995029 11124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:40:26.995628 11124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:40:26 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:40:27.018482 11124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "8a1355d456074ea0a92eaa1c161e41f8"
format_stamp: "Formatted at 2026-08-12 16:40:27 on dist-test-slave-f7th"
server_key: "7d26b25e6f96b551ebe4e76a4913b10f"
server_key_iv: "849c5aafc975999b6be9b44f333fc343"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:27.019886 11124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "8a1355d456074ea0a92eaa1c161e41f8"
format_stamp: "Formatted at 2026-08-12 16:40:27 on dist-test-slave-f7th"
server_key: "7d26b25e6f96b551ebe4e76a4913b10f"
server_key_iv: "849c5aafc975999b6be9b44f333fc343"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:27.038035 11124 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.012s
I20260812 16:40:27.061349 11144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:40:27.063241 11124 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.008s sys 0.000s
I20260812 16:40:27.063488 11124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "8a1355d456074ea0a92eaa1c161e41f8"
format_stamp: "Formatted at 2026-08-12 16:40:27 on dist-test-slave-f7th"
server_key: "7d26b25e6f96b551ebe4e76a4913b10f"
server_key_iv: "849c5aafc975999b6be9b44f333fc343"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:27.063860 11124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:40:27.099756 11124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:40:27.104329 11124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:40:27.104647 11124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:40:27.105964 11124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:40:27.111133 11124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:40:27.111235 11124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:40:27.111399 11124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:40:27.111585 11124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 16:40:27 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:40:27.279575 11124 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:45603
I20260812 16:40:27.279690 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:45603 every 8 connection(s)
I20260812 16:40:27.281440 11124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:40:27.287497 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 11124
I20260812 16:40:27.287745 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:40:27.288827 31990 external_mini_cluster.cc:1428] Setting key 570c987445bc9f7bc1cecd4063399b25
Aug 12 16:40:27 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:40:27.307981 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:27.284883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:49601 (local address 127.31.61.190:43723)
0812 16:40:27.285421 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:27.285428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:27.286612 (+ 1184us) server_negotiation.cc:407] Connection header received
0812 16:40:27.288622 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:27.288628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:27.288722 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:27.288822 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:40:27.290944 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:27.292395 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:27.294836 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:27.295655 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:27.295781 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:27.300713 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:40:27.300720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:40:27.300724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:40:27.300766 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:40:27.304256 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:27.305073 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:27.305078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:27.305081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:27.305266 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:27.305820 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:27.305825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:27.305827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:27.306201 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:40:27.306335 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:27.307140 (+ 805us) server_negotiation.cc:299] Negotiation successful
0812 16:40:27.307370 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":250,"threads_started":1}
I20260812 16:40:27.309111 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:27.285500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43723 (local address 127.31.61.129:49601)
0812 16:40:27.286416 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0812 16:40:27.286463 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 16:40:27.288331 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:40:27.289131 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:40:27.289140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:40:27.289715 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:40:27.290735 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:27.290764 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:27.292607 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:27.292613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:27.294662 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:27.294680 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:27.295899 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:27.295905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:27.296357 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:27.297269 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:40:27.297298 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:40:27.300346 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:40:27.304435 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:40:27.304446 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:40:27.304463 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:40:27.304940 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:40:27.305400 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:40:27.305404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:40:27.305406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:40:27.305700 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:40:27.306348 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:40:27.306357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:40:27.306732 (+ 375us) client_negotiation.cc:769] Sending connection context
0812 16:40:27.307038 (+ 306us) client_negotiation.cc:240] Negotiation successful
0812 16:40:27.307659 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":404,"threads_started":1}
I20260812 16:40:27.310686 11258 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43723
I20260812 16:40:27.310995 11258 heartbeater.cc:464] Registering TS with master...
I20260812 16:40:27.311888 11258 heartbeater.cc:511] Master 127.31.61.190:43723 requested a full tablet report, sending...
I20260812 16:40:27.314265 11042 ts_manager.cc:194] Registered new tserver with Master: 8a1355d456074ea0a92eaa1c161e41f8 (127.31.61.129:45603)
I20260812 16:40:27.316262 11042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:49601
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:40:27.364147 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:52697/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43723
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:40:27 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
W20260812 16:40:27.975993 11264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:40:27.976397 11264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:40:27.976583 11264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:40:27.987491 11264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:40:27.987803 11264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:40:27.988052 11264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:40:28.000692 11264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.61.148:52697/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:40:28.002208 11264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:40:28.003779 11264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:40:28.021451 11278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:40:28.021901 11279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:40:28.022194 11281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:40:28.023411 11264 server_base.cc:1034] running on GCE node
I20260812 16:40:28.024492 11264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:40:28.025586 11264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:40:28.026927 11264 hybrid_clock.cc:648] HybridClock initialized: now 1786552828026799 us; error 106 us; skew 500 ppm
I20260812 16:40:28.027719 11264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:40:28 dist-test-slave-f7th krb5kdc[10558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552828, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:40:28.032202 11264 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:40:28.034173 11264 webserver.cc:460] Webserver started at http://127.31.61.130:43591/ using document root <none> and password file <none>
I20260812 16:40:28.035068 11264 fs_manager.cc:359] Metadata directory not provided
I20260812 16:40:28.035211 11264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:40:28.035667 11264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:40:28 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552828, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:40:28.054833 11264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb"
format_stamp: "Formatted at 2026-08-12 16:40:28 on dist-test-slave-f7th"
server_key: "3bef99ace13e6d2a730dca4917d00898"
server_key_iv: "02d302e4c8cafe27865b30e797e4687e"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:28.056047 11264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb"
format_stamp: "Formatted at 2026-08-12 16:40:28 on dist-test-slave-f7th"
server_key: "3bef99ace13e6d2a730dca4917d00898"
server_key_iv: "02d302e4c8cafe27865b30e797e4687e"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:28.063433 11264 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 16:40:28.083376 11288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:40:28.085273 11264 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 16:40:28.085458 11264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb"
format_stamp: "Formatted at 2026-08-12 16:40:28 on dist-test-slave-f7th"
server_key: "3bef99ace13e6d2a730dca4917d00898"
server_key_iv: "02d302e4c8cafe27865b30e797e4687e"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:28.085752 11264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:40:28.126284 11264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:40:28.131146 11264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:40:28.131731 11264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:40:28.133373 11264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:40:28.135301 11264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:40:28.135406 11264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:40:28.135711 11264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:40:28.135803 11264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 16:40:28.191900 11264 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:37449
I20260812 16:40:28.191968 11402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:37449 every 8 connection(s)
I20260812 16:40:28.193169 11264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:40:28.193672 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 11264
I20260812 16:40:28.193842 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:40:28.194319 31990 external_mini_cluster.cc:1428] Setting key 11c5b386cb1447005927e0633dfa22b2
Aug 12 16:40:28 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552828, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:40:28.217875 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.196418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:55665 (local address 127.31.61.190:43723)
0812 16:40:28.197075 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:28.197081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:28.198408 (+ 1327us) server_negotiation.cc:407] Connection header received
0812 16:40:28.200994 (+ 2586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:28.201000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:28.201126 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:28.201249 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:40:28.202954 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.204212 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.205936 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.206507 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.206652 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:28.211192 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:40:28.211198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:40:28.211203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:40:28.211286 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:40:28.214493 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:28.215377 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:28.215382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:28.215384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:28.215514 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:28.215981 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:28.215985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:28.215987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:28.216285 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:40:28.216420 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:28.217272 (+ 852us) server_negotiation.cc:299] Negotiation successful
0812 16:40:28.217511 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":387,"threads_started":1}
I20260812 16:40:28.218850 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.197345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43723 (local address 127.31.61.130:55665)
0812 16:40:28.198238 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0812 16:40:28.198292 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 16:40:28.200686 (+ 2394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:40:28.201504 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:40:28.201515 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:40:28.202054 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:40:28.202771 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:28.202790 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.204364 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.204369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:28.205800 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:28.205815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.206753 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.206759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:28.207154 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:28.208085 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:40:28.208126 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:40:28.210924 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:40:28.214666 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:40:28.214676 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:40:28.214694 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:40:28.215258 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:40:28.215641 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:40:28.215647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:40:28.215650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:40:28.215867 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:40:28.216414 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:40:28.216422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:40:28.216881 (+ 459us) client_negotiation.cc:769] Sending connection context
0812 16:40:28.217137 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 16:40:28.217740 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":381,"threads_started":1}
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
I20260812 16:40:28.220424 11403 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43723
I20260812 16:40:28.220835 11403 heartbeater.cc:464] Registering TS with master...
I20260812 16:40:28.221728 11403 heartbeater.cc:511] Master 127.31.61.190:43723 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
I20260812 16:40:28.223430 11042 ts_manager.cc:194] Registered new tserver with Master: 3f32bcb9d67d4aa29d551b1f2469e8bb (127.31.61.130:37449)
I20260812 16:40:28.224242 11042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:55665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:40:28.252023 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:52697/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43723
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:40:28.320797 11258 heartbeater.cc:503] Master 127.31.61.190:43723 was elected leader, sending a full tablet report...
W20260812 16:40:28.722290 11409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:40:28.722627 11409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:40:28.722781 11409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:40:28.731779 11409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:40:28.731977 11409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:40:28.732187 11409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:40:28.744557 11409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.61.148:52697/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:40:28.746090 11409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:40:28.747545 11409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:40:28.763486 11415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:40:28.763677 11416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:40:28.763701 11418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:40:28.765216 11409 server_base.cc:1034] running on GCE node
I20260812 16:40:28.765893 11409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:40:28.766953 11409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:40:28.768165 11409 hybrid_clock.cc:648] HybridClock initialized: now 1786552828768113 us; error 45 us; skew 500 ppm
I20260812 16:40:28.768607 11409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:40:28 dist-test-slave-f7th krb5kdc[10558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552828, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:40:28.772943 11409 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:40:28.774792 11409 webserver.cc:460] Webserver started at http://127.31.61.131:45003/ using document root <none> and password file <none>
I20260812 16:40:28.775537 11409 fs_manager.cc:359] Metadata directory not provided
I20260812 16:40:28.775691 11409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:40:28.776008 11409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:40:28 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552828, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:40:28.795166 11409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "a00fbd3830c14baa9fd826dea7d158f2"
format_stamp: "Formatted at 2026-08-12 16:40:28 on dist-test-slave-f7th"
server_key: "a3a9998b0078e3d6e77f61936c70c424"
server_key_iv: "f33e4c9cc26a1155c82fe27772d2b04c"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:28.796746 11409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "a00fbd3830c14baa9fd826dea7d158f2"
format_stamp: "Formatted at 2026-08-12 16:40:28 on dist-test-slave-f7th"
server_key: "a3a9998b0078e3d6e77f61936c70c424"
server_key_iv: "f33e4c9cc26a1155c82fe27772d2b04c"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:28.805541 11409 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260812 16:40:28.824455 11425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:40:28.826051 11409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 16:40:28.826223 11409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "a00fbd3830c14baa9fd826dea7d158f2"
format_stamp: "Formatted at 2026-08-12 16:40:28 on dist-test-slave-f7th"
server_key: "a3a9998b0078e3d6e77f61936c70c424"
server_key_iv: "f33e4c9cc26a1155c82fe27772d2b04c"
server_key_version: "kuduclusterkey@0"
I20260812 16:40:28.826422 11409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:40:28.848961 11409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:40:28.854029 11409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:40:28.854367 11409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:40:28.855472 11409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:40:28.857191 11409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:40:28.857264 11409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:40:28.857358 11409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:40:28.857401 11409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:40:28.914721 11409 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:41327
I20260812 16:40:28.914855 11539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:41327 every 8 connection(s)
I20260812 16:40:28.916229 11409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:40:28.919986 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 11409
I20260812 16:40:28.920169 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:40:28.920684 31990 external_mini_cluster.cc:1428] Setting key 8983b3a12a52c9fccd554bb9465aee0e
Aug 12 16:40:28 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552828, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:40:28.940244 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.919209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:57767 (local address 127.31.61.190:43723)
0812 16:40:28.919865 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:28.919872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:28.921904 (+ 2032us) server_negotiation.cc:407] Connection header received
0812 16:40:28.924454 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:28.924458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:28.924549 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:28.924774 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:40:28.926260 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.927077 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.928738 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.929354 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.929461 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:28.933325 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:40:28.933334 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:40:28.933337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:40:28.933370 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:40:28.936936 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:28.937721 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:28.937725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:28.937727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:28.937896 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:28.938345 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:28.938348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:28.938351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:28.938650 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:40:28.938783 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:28.939573 (+ 790us) server_negotiation.cc:299] Negotiation successful
0812 16:40:28.939872 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":404,"threads_started":1}
I20260812 16:40:28.941325 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.920105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43723 (local address 127.31.61.131:57767)
0812 16:40:28.921619 (+ 1514us) negotiation.cc:107] Waiting for socket to connect
0812 16:40:28.921659 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 16:40:28.924165 (+ 2506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:40:28.924987 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:40:28.924997 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:40:28.925476 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:40:28.926094 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:28.926117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.927199 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.927202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:28.928600 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:28.928620 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.929579 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.929584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:28.929859 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:28.930723 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:40:28.930747 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:40:28.933097 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:40:28.937100 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:40:28.937108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:40:28.937122 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:40:28.937621 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:40:28.938014 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:40:28.938018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:40:28.938021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:40:28.938249 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:40:28.938796 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:40:28.938806 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:40:28.939187 (+ 381us) client_negotiation.cc:769] Sending connection context
0812 16:40:28.939477 (+ 290us) client_negotiation.cc:240] Negotiation successful
0812 16:40:28.940059 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"thread_start_us":578,"threads_started":1}
I20260812 16:40:28.942698 11540 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43723
I20260812 16:40:28.942982 11540 heartbeater.cc:464] Registering TS with master...
I20260812 16:40:28.943904 11540 heartbeater.cc:511] Master 127.31.61.190:43723 requested a full tablet report, sending...
I20260812 16:40:28.945343 11042 ts_manager.cc:194] Registered new tserver with Master: a00fbd3830c14baa9fd826dea7d158f2 (127.31.61.131:41327)
I20260812 16:40:28.946338 11042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:57767
I20260812 16:40:28.949723 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:40:28.969115 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.954478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.31.61.190:43723)
0812 16:40:28.954641 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:28.954645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:28.955401 (+ 756us) server_negotiation.cc:407] Connection header received
0812 16:40:28.955626 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:28.955632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:28.955741 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:28.955862 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:40:28.957535 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.958481 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.960216 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:28.960812 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:28.960905 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:28.963936 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:40:28.963941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:40:28.963944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:40:28.963976 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:40:28.966820 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:28.967483 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:28.967486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:28.967489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:28.967661 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:28.967969 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:28.967972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:28.967974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:28.968230 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:40:28.968405 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:28.968668 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 16:40:28.968838 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:40:28.973809 11042 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:40:28.982059 11042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:40:29.016511 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.996223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.31.61.129:45603)
0812 16:40:28.997504 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:28.997514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:29.004189 (+ 6675us) server_negotiation.cc:407] Connection header received
0812 16:40:29.004662 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:29.004672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:29.004948 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:29.005371 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:40:29.006836 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.008827 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.013056 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.014244 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.014847 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.015125 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:29.015875 (+ 750us) server_negotiation.cc:299] Negotiation successful
0812 16:40:29.016071 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":853,"threads_started":1}
I20260812 16:40:29.016880 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.996011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:45603 (local address 127.0.0.1:39138)
0812 16:40:29.003858 (+ 7847us) negotiation.cc:107] Waiting for socket to connect
0812 16:40:29.003904 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 16:40:29.004384 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:40:29.005934 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:40:29.005943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:40:29.005978 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:40:29.006601 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.006615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.009050 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.009056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.012907 (+ 3851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.012920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.015290 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.015297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.015511 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.015572 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 16:40:29.015794 (+ 222us) client_negotiation.cc:240] Negotiation successful
0812 16:40:29.015991 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7593,"thread_start_us":7318,"threads_started":1}
I20260812 16:40:29.020071 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.996868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.31.61.131:41327)
0812 16:40:28.998133 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:28.998141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:29.010179 (+ 12038us) server_negotiation.cc:407] Connection header received
0812 16:40:29.010355 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:29.010362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:29.010634 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:29.010854 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:40:29.012309 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.014053 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.016884 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.018374 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.018696 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.018935 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:29.019487 (+ 552us) server_negotiation.cc:299] Negotiation successful
0812 16:40:29.019680 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":884,"threads_started":1}
I20260812 16:40:29.020871 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.997780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:41327 (local address 127.0.0.1:55542)
0812 16:40:29.010059 (+ 12279us) negotiation.cc:107] Waiting for socket to connect
0812 16:40:29.010078 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:40:29.010205 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:40:29.011280 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:40:29.011285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:40:29.011314 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:40:29.012135 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.012146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.014206 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.014211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.016683 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.016699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.019091 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.019097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.019280 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.019315 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:40:29.020269 (+ 954us) client_negotiation.cc:240] Negotiation successful
0812 16:40:29.020414 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12140,"mutex_wait_us":9,"thread_start_us":12863,"threads_started":1}
I20260812 16:40:29.021862 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.998630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.31.61.130:37449)
0812 16:40:29.009068 (+ 10438us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:29.009078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:29.011038 (+ 1960us) server_negotiation.cc:407] Connection header received
0812 16:40:29.011197 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:29.011204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:29.011472 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:29.011722 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:40:29.012679 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.014399 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.018318 (+ 3919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.019829 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.020041 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.021296 (+ 1255us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:29.021352 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 16:40:29.021482 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10256,"thread_start_us":10082,"threads_started":1}
I20260812 16:40:29.022439 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:28.998406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:37449 (local address 127.0.0.1:39150)
0812 16:40:29.010897 (+ 12491us) negotiation.cc:107] Waiting for socket to connect
0812 16:40:29.010928 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 16:40:29.011048 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:40:29.011719 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:40:29.011724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:40:29.011749 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:40:29.012462 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.012477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.015696 (+ 3219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.015702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.018144 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.018158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.020517 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.020528 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.020745 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.020783 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:40:29.020870 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 16:40:29.022097 (+ 1227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12369,"mutex_wait_us":431,"thread_start_us":11035,"threads_started":1}
I20260812 16:40:29.024748 11473 tablet_service.cc:1467] Processing CreateTablet for tablet 9257a4c476cc49fb883a00caa166b3fb (DEFAULT_TABLE table=test-table [id=6c3e4e4d6ffe49cb8206611c13a98eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:40:29.026202 11473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9257a4c476cc49fb883a00caa166b3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:40:29.026108 11337 tablet_service.cc:1467] Processing CreateTablet for tablet 9257a4c476cc49fb883a00caa166b3fb (DEFAULT_TABLE table=test-table [id=6c3e4e4d6ffe49cb8206611c13a98eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:40:29.027299 11337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9257a4c476cc49fb883a00caa166b3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:40:29.021327 11193 tablet_service.cc:1467] Processing CreateTablet for tablet 9257a4c476cc49fb883a00caa166b3fb (DEFAULT_TABLE table=test-table [id=6c3e4e4d6ffe49cb8206611c13a98eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:40:29.031513 11193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9257a4c476cc49fb883a00caa166b3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:40:29.041191 11559 tablet_bootstrap.cc:492] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: Bootstrap starting.
I20260812 16:40:29.044771 11560 tablet_bootstrap.cc:492] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: Bootstrap starting.
I20260812 16:40:29.045390 11559 tablet_bootstrap.cc:654] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: Neither blocks nor log segments found. Creating new log.
I20260812 16:40:29.047394 11559 log.cc:824] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: Log is configured to *not* fsync() on all Append() calls
I20260812 16:40:29.049604 11560 tablet_bootstrap.cc:654] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: Neither blocks nor log segments found. Creating new log.
I20260812 16:40:29.051254 11560 log.cc:824] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: Log is configured to *not* fsync() on all Append() calls
I20260812 16:40:29.051582 11559 tablet_bootstrap.cc:492] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: No bootstrap required, opened a new log
I20260812 16:40:29.051915 11559 ts_tablet_manager.cc:1397] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260812 16:40:29.054653 11561 tablet_bootstrap.cc:492] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: Bootstrap starting.
I20260812 16:40:29.058938 11561 tablet_bootstrap.cc:654] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: Neither blocks nor log segments found. Creating new log.
I20260812 16:40:29.060323 11561 log.cc:824] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: Log is configured to *not* fsync() on all Append() calls
I20260812 16:40:29.061388 11560 tablet_bootstrap.cc:492] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: No bootstrap required, opened a new log
I20260812 16:40:29.061683 11560 ts_tablet_manager.cc:1397] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20260812 16:40:29.063311 11561 tablet_bootstrap.cc:492] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: No bootstrap required, opened a new log
I20260812 16:40:29.063102 11559 raft_consensus.cc:348] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } }
I20260812 16:40:29.063565 11559 raft_consensus.cc:374] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:40:29.063695 11559 raft_consensus.cc:729] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f32bcb9d67d4aa29d551b1f2469e8bb, State: Initialized, Role: FOLLOWER
I20260812 16:40:29.063766 11561 ts_tablet_manager.cc:1397] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.007s
I20260812 16:40:29.064505 11559 consensus_queue.cc:260] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } }
I20260812 16:40:29.066059 11403 heartbeater.cc:503] Master 127.31.61.190:43723 was elected leader, sending a full tablet report...
I20260812 16:40:29.066910 11559 ts_tablet_manager.cc:1428] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260812 16:40:29.074306 11560 raft_consensus.cc:348] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } }
I20260812 16:40:29.074262 11561 raft_consensus.cc:348] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } }
I20260812 16:40:29.074688 11560 raft_consensus.cc:374] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:40:29.074765 11561 raft_consensus.cc:374] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:40:29.074846 11560 raft_consensus.cc:729] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00fbd3830c14baa9fd826dea7d158f2, State: Initialized, Role: FOLLOWER
I20260812 16:40:29.074894 11561 raft_consensus.cc:729] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1355d456074ea0a92eaa1c161e41f8, State: Initialized, Role: FOLLOWER
I20260812 16:40:29.075496 11561 consensus_queue.cc:260] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } }
I20260812 16:40:29.075645 11560 consensus_queue.cc:260] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } }
I20260812 16:40:29.076985 11540 heartbeater.cc:503] Master 127.31.61.190:43723 was elected leader, sending a full tablet report...
I20260812 16:40:29.077477 11560 ts_tablet_manager.cc:1428] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260812 16:40:29.077733 11561 ts_tablet_manager.cc:1428] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260812 16:40:29.168751 11541 tablet.cc:2369] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:40:29.196038 11404 tablet.cc:2369] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:40:29.286151 11259 tablet.cc:2369] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:40:29.334880 11567 raft_consensus.cc:482] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:40:29.335213 11567 raft_consensus.cc:504] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } }
I20260812 16:40:29.337052 11567 leader_election.cc:288] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a1355d456074ea0a92eaa1c161e41f8 (127.31.61.129:45603), 3f32bcb9d67d4aa29d551b1f2469e8bb (127.31.61.130:37449)
I20260812 16:40:29.343690 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:29.337331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:37449 (local address 127.31.61.131:59999)
0812 16:40:29.337624 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 16:40:29.337645 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:40:29.337767 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:40:29.338238 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:40:29.338242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:40:29.338276 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:40:29.338477 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.338497 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.340014 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.340019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.341618 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.341633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.342914 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.342919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.343113 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.343138 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 16:40:29.343219 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 16:40:29.343351 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":1}
I20260812 16:40:29.343696 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:29.337665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:59999 (local address 127.31.61.130:37449)
0812 16:40:29.337910 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:29.337916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:29.337935 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:40:29.338019 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:29.338023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:29.338113 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:29.338195 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:40:29.338933 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.339868 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.341777 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.342772 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.342913 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.342998 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:29.343259 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 16:40:29.343402 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 16:40:29.344879 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:29.337740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:38879 (local address 127.31.61.129:45603)
0812 16:40:29.338049 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:29.338055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:29.338485 (+ 430us) server_negotiation.cc:407] Connection header received
0812 16:40:29.338652 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:29.338656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:29.338716 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:29.338780 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:40:29.339635 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.340617 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.342591 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.343722 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.343891 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.343966 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:29.344489 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 16:40:29.344621 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260812 16:40:29.345065 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:29.337423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:45603 (local address 127.31.61.131:38879)
0812 16:40:29.338370 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0812 16:40:29.338385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:40:29.338515 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:40:29.338804 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:40:29.338807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:40:29.338825 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:40:29.339468 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.339480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.340769 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.340772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.342430 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:40:29.342443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.344096 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.344104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:40:29.344322 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.344356 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:40:29.344433 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 16:40:29.344528 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":22,"thread_start_us":588,"threads_started":1}
I20260812 16:40:29.344964 11357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9257a4c476cc49fb883a00caa166b3fb" candidate_uuid: "a00fbd3830c14baa9fd826dea7d158f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" is_pre_election: true
I20260812 16:40:29.345738 11357 raft_consensus.cc:2454] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a00fbd3830c14baa9fd826dea7d158f2 in term 0.
I20260812 16:40:29.346333 11213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9257a4c476cc49fb883a00caa166b3fb" candidate_uuid: "a00fbd3830c14baa9fd826dea7d158f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1355d456074ea0a92eaa1c161e41f8" is_pre_election: true
I20260812 16:40:29.346764 11429 leader_election.cc:302] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f32bcb9d67d4aa29d551b1f2469e8bb, a00fbd3830c14baa9fd826dea7d158f2; no voters:
I20260812 16:40:29.346951 11213 raft_consensus.cc:2454] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a00fbd3830c14baa9fd826dea7d158f2 in term 0.
I20260812 16:40:29.347249 11567 raft_consensus.cc:2781] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:40:29.347402 11567 raft_consensus.cc:482] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:40:29.347556 11567 raft_consensus.cc:3037] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:40:29.350682 11567 raft_consensus.cc:504] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } }
I20260812 16:40:29.351588 11567 leader_election.cc:288] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8a1355d456074ea0a92eaa1c161e41f8 (127.31.61.129:45603), 3f32bcb9d67d4aa29d551b1f2469e8bb (127.31.61.130:37449)
I20260812 16:40:29.352108 11213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9257a4c476cc49fb883a00caa166b3fb" candidate_uuid: "a00fbd3830c14baa9fd826dea7d158f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1355d456074ea0a92eaa1c161e41f8"
I20260812 16:40:29.352324 11357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9257a4c476cc49fb883a00caa166b3fb" candidate_uuid: "a00fbd3830c14baa9fd826dea7d158f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb"
I20260812 16:40:29.352372 11213 raft_consensus.cc:3037] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:40:29.352655 11357 raft_consensus.cc:3037] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:40:29.356043 11213 raft_consensus.cc:2454] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a00fbd3830c14baa9fd826dea7d158f2 in term 1.
I20260812 16:40:29.356381 11357 raft_consensus.cc:2454] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a00fbd3830c14baa9fd826dea7d158f2 in term 1.
I20260812 16:40:29.356693 11428 leader_election.cc:302] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1355d456074ea0a92eaa1c161e41f8, a00fbd3830c14baa9fd826dea7d158f2; no voters:
I20260812 16:40:29.357168 11567 raft_consensus.cc:2781] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:40:29.357842 11567 raft_consensus.cc:686] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 1 LEADER]: Becoming Leader. State: Replica: a00fbd3830c14baa9fd826dea7d158f2, State: Running, Role: LEADER
I20260812 16:40:29.358387 11567 consensus_queue.cc:237] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } }
I20260812 16:40:29.363276 11042 catalog_manager.cc:5495] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a00fbd3830c14baa9fd826dea7d158f2 (127.31.61.131). New cstate: current_term: 1 leader_uuid: "a00fbd3830c14baa9fd826dea7d158f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a00fbd3830c14baa9fd826dea7d158f2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 } health_report { overall_health: UNKNOWN } } }
I20260812 16:40:29.388286 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:29.381509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.31.61.131:41327)
0812 16:40:29.381778 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:29.381785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:29.381888 (+ 103us) server_negotiation.cc:407] Connection header received
0812 16:40:29.382101 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:29.382105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:29.382185 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:29.382322 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:40:29.382754 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.383835 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.386114 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.386533 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.386655 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.387171 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:40:29.387761 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:40:29.387882 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:29.387984 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 16:40:29.388052 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 16:40:29.398927 11357 raft_consensus.cc:1261] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [term 1 FOLLOWER]: Refusing update from remote peer a00fbd3830c14baa9fd826dea7d158f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:40:29.398944 11213 raft_consensus.cc:1261] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [term 1 FOLLOWER]: Refusing update from remote peer a00fbd3830c14baa9fd826dea7d158f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:40:29.400123 11569 consensus_queue.cc:1035] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1355d456074ea0a92eaa1c161e41f8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:40:29.400508 11567 consensus_queue.cc:1035] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f32bcb9d67d4aa29d551b1f2469e8bb" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:40:29.413273 11572 mvcc.cc:205] Tried to move back new op lower bound from 7317720389207351296 to 7317720389056266240. Current Snapshot: MvccSnapshot[applied={T|T < 7317720389207351296}]
I20260812 16:40:29.414026 11573 mvcc.cc:205] Tried to move back new op lower bound from 7317720389207351296 to 7317720389056266240. Current Snapshot: MvccSnapshot[applied={T|T < 7317720389207351296}]
I20260812 16:40:29.427222 11041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:40:29.427523 11041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:40:29.431419 11041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93f51e0ae45942e0a76225ed8e769637: Sending DeleteTablet for 3 replicas of tablet 9257a4c476cc49fb883a00caa166b3fb
I20260812 16:40:29.432621 11193 tablet_service.cc:1514] Processing DeleteTablet for tablet 9257a4c476cc49fb883a00caa166b3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:40:29 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:39138
I20260812 16:40:29.432617 11337 tablet_service.cc:1514] Processing DeleteTablet for tablet 9257a4c476cc49fb883a00caa166b3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:40:29 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:39150
I20260812 16:40:29.433548 11582 tablet_replica.cc:330] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: stopping tablet replica
I20260812 16:40:29.434033 11473 tablet_service.cc:1514] Processing DeleteTablet for tablet 9257a4c476cc49fb883a00caa166b3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:40:29 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:55542
I20260812 16:40:29.434046 11582 raft_consensus.cc:2229] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:40:29.434576 11582 raft_consensus.cc:2258] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:40:29.435089 11583 tablet_replica.cc:330] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: stopping tablet replica
I20260812 16:40:29.435336 11581 tablet_replica.cc:330] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: stopping tablet replica
I20260812 16:40:29.435581 11583 raft_consensus.cc:2229] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:40:29.435777 11581 raft_consensus.cc:2229] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:40:29.436347 11581 raft_consensus.cc:2258] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:40:29.436357 11583 raft_consensus.cc:2258] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:40:29.437808 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 11124
I20260812 16:40:29.438546 11581 ts_tablet_manager.cc:1905] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:40:29.439360 11583 ts_tablet_manager.cc:1905] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:40:29.441710 11582 ts_tablet_manager.cc:1905] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:40:29.447286 11581 ts_tablet_manager.cc:1918] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:40:29.447427 11583 ts_tablet_manager.cc:1918] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:40:29.447670 11583 log.cc:1196] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/9257a4c476cc49fb883a00caa166b3fb
I20260812 16:40:29.448256 11583 ts_tablet_manager.cc:1939] T 9257a4c476cc49fb883a00caa166b3fb P a00fbd3830c14baa9fd826dea7d158f2: Deleting consensus metadata
I20260812 16:40:29.450230 11027 catalog_manager.cc:4841] TS a00fbd3830c14baa9fd826dea7d158f2 (127.31.61.131:41327): tablet 9257a4c476cc49fb883a00caa166b3fb (table test-table [id=6c3e4e4d6ffe49cb8206611c13a98eac]) successfully deleted
I20260812 16:40:29.454460 11581 log.cc:1196] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/9257a4c476cc49fb883a00caa166b3fb
I20260812 16:40:29.454574 11582 ts_tablet_manager.cc:1918] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:40:29.454826 11582 log.cc:1196] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/9257a4c476cc49fb883a00caa166b3fb
I20260812 16:40:29.455169 11581 ts_tablet_manager.cc:1939] T 9257a4c476cc49fb883a00caa166b3fb P 3f32bcb9d67d4aa29d551b1f2469e8bb: Deleting consensus metadata
I20260812 16:40:29.455417 11582 ts_tablet_manager.cc:1939] T 9257a4c476cc49fb883a00caa166b3fb P 8a1355d456074ea0a92eaa1c161e41f8: Deleting consensus metadata
I20260812 16:40:29.457240 11029 catalog_manager.cc:4841] TS 3f32bcb9d67d4aa29d551b1f2469e8bb (127.31.61.130:37449): tablet 9257a4c476cc49fb883a00caa166b3fb (table test-table [id=6c3e4e4d6ffe49cb8206611c13a98eac]) successfully deleted
I20260812 16:40:29.457295 11028 catalog_manager.cc:4841] TS 8a1355d456074ea0a92eaa1c161e41f8 (127.31.61.129:45603): tablet 9257a4c476cc49fb883a00caa166b3fb (table test-table [id=6c3e4e4d6ffe49cb8206611c13a98eac]) successfully deleted
Aug 12 16:40:29 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:40:29.465281 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:29.439174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.31.61.129:45603)
0812 16:40:29.439354 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:29.439361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:29.448455 (+ 9094us) server_negotiation.cc:407] Connection header received
0812 16:40:29.448992 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:29.448998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:29.449098 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:29.449203 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:40:29.451453 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.452272 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.453579 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.454035 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.454150 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.459225 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:40:29.459236 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:40:29.459252 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:40:29.459284 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:40:29.462122 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:29.462783 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:29.462790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:29.462792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:29.462926 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:29.463306 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:29.463310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:29.463312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:29.463838 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:40:29.464140 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:29.464726 (+ 586us) server_negotiation.cc:299] Negotiation successful
0812 16:40:29.464943 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 16:40:29.465992 11253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:40:29.558637 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 11124
I20260812 16:40:29.583514 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 11264
Aug 12 16:40:29 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:40:29.601200 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:29.584388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.31.61.130:37449)
0812 16:40:29.584562 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:29.584566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:29.584609 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:40:29.584748 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:29.584752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:29.584814 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:29.584895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:40:29.586648 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.587347 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.589289 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.589792 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.589904 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.595344 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:40:29.595361 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:40:29.595378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:40:29.595452 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:40:29.598329 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:29.599165 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:29.599173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:29.599176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:29.599324 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:29.599737 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:29.599740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:29.599743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:29.600151 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:40:29.600367 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:29.600720 (+ 353us) server_negotiation.cc:299] Negotiation successful
0812 16:40:29.600892 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:40:29.601912 11398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:40:29.692464 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 11264
I20260812 16:40:29.708843 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 11409
Aug 12 16:40:29 dist-test-slave-f7th krb5kdc[10558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:40:29.725167 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:40:29.709670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.31.61.131:41327)
0812 16:40:29.709884 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:40:29.709889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:40:29.709914 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:40:29.710031 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:40:29.710037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:40:29.710103 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:40:29.710199 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:40:29.711477 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.712316 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.713553 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:40:29.714007 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:40:29.714131 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:40:29.718975 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:40:29.718988 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:40:29.719003 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:40:29.719048 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:40:29.722043 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:29.722948 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:29.722955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:29.722958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:29.723117 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:40:29.723537 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:40:29.723541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:40:29.723543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:40:29.724016 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:40:29.724271 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 16:40:29.724678 (+ 407us) server_negotiation.cc:299] Negotiation successful
0812 16:40:29.724848 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 16:40:29.725912 11535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:40:29.847404 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 11409
I20260812 16:40:29.864739 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 11010
I20260812 16:40:29.865875 11073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:40:29.979558 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 11010
I20260812 16:40:29.998919 31990 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 16:40:30.334805 31990 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 16:40:30.335139 31990 mini_ranger.cc:66] Stopping Ranger...
I20260812 16:40:30.671339 31990 mini_ranger.cc:68] Stopped Ranger
2026-08-12 16:40:30.671 UTC [10590] LOG: received smart shutdown request
2026-08-12 16:40:30.676 UTC [10590] LOG: background worker "logical replication launcher" (PID 10599) exited with exit code 1
2026-08-12 16:40:30.676 UTC [10594] LOG: shutting down
2026-08-12 16:40:30.896 UTC [10590] LOG: database system is shut down
2026-08-12T16:40:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53752 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:40:31 dist-test-slave-f7th krb5kdc[11597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:40:31 dist-test-slave-f7th krb5kdc[11597](info): set up 2 sockets
Aug 12 16:40:31 dist-test-slave-f7th krb5kdc[11597](info): commencing operation
krb5kdc: starting...
W20260812 16:40:32.071381 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:40:32 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:40:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:40:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.61.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.61.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/rangeradmin_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/rangeradmin_127.31.61.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.61.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.61.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/rangerlookup_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/rangerlookup_127.31.61.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.61.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.61.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/HTTP_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/HTTP_127.31.61.148@KRBTEST.COM.keytab.
I20260812 16:40:32.431046 31990 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgT9LAJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/postgres -l logfile start
2026-08-12 16:40:35.149 UTC [11628] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 16:40:35.149 UTC [11628] LOG: listening on IPv4 address "127.31.61.148", port 57397
2026-08-12 16:40:35.155 UTC [11628] LOG: listening on Unix socket "/tmp/.s.PGSQL.57397"
2026-08-12 16:40:35.179 UTC [11631] LOG: database system was shut down at 2026-08-12 16:40:33 UTC
2026-08-12 16:40:35.183 UTC [11628] LOG: database system is ready to accept connections
I20260812 16:40:36.091029 31990 mini_postgres.cc:96] Postgres bound to 57397
2026-08-12 16:40:36.113 UTC [11639] FATAL: database "slave" does not exist
127.31.61.148:57397 - accepting connections
I20260812 16:40:36.113636 31990 mini_ranger.cc:161] Starting Ranger...
I20260812 16:40:36.141434 31990 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 16:40:36.686205 31990 mini_ranger.cc:87] Created ranger Postgres database
I20260812 16:40:36.686338 31990 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-admin
2026-08-12 16:40:36,978 [I] DB FLAVOR :POSTGRES
2026-08-12 16:40:36,978 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 16:40:36,978 [I] Checking connection..
2026-08-12 16:40:36,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 16:40:37,242 [I] Checking connection passed.
2026-08-12 16:40:37,243 [I] --------- Verifying version history table ---------
2026-08-12 16:40:37,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:40:37,494 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 16:40:37,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:40:37,760 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 16:40:37,761 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 16:40:37,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 16:40:38.018 UTC [11703] WARNING: there is no transaction in progress
2026-08-12 16:40:38,025 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 16:40:38,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 16:40:38,285 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 16:40:38,285 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 16:40:38,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 16:40:38,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 16:40:38,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f7th.c.gcp-upstream.internal','N') ;"
2026-08-12 16:40:39,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 16:40:39,272 [I] Table x_portal_user does not exist in database ranger
2026-08-12 16:40:39,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 16:40:39,523 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 16:40:39,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 16:40:39,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 16:40:40,036 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 16:40:40,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 16:40:41.105 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.124 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.146 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.166 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.185 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.206 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.274 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.282 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.297 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.309 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.320 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.335 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.345 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.353 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.361 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:41.370 UTC [11811] WARNING: there is no transaction in progress
2026-08-12 16:40:42,127 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 16:40:42,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 16:40:42,382 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 16:40:42,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 16:40:42,640 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 16:40:42,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 16:40:42,891 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 16:40:42,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 16:40:43,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 16:40:43,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f7th.c.gcp-upstream.internal';"
2026-08-12 16:40:43,639 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 16:40:43,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 16:40:43,887 [I] DB_PATCHES have already been applied
I20260812 16:40:43.892918 31990 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgT9LAJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 16:40:43.893041 31990 mini_ranger.cc:193] Using host: 127.31.61.148
I20260812 16:40:43.908129 31990 mini_ranger.cc:227] Ranger admin URL: http://127.31.61.148:42543
Aug 12, 2026 4:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 4:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 4:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 4:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 4:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 4:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 4:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 16:40:44.949735 31990 mini_ranger.cc:160] Time spent starting Ranger: real 8.836s user 0.008s sys 0.037s
Aug 12, 2026 4:41:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 4:41:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 12, 2026 4:41:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 4:41:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 4:41:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 16:41:06.343024 31990 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.61.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.61.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/rangerkms_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.61.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/rangerkms_127.31.61.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.61.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.61.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.61.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/HTTP_127.31.61.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.61.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/HTTP_127.31.61.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 16:41:06 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 16:41:06.549146 31990 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 16:41:06.574404 31990 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 16:41:07.638391 31990 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 16:41:07.638496 31990 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-kms
I20260812 16:41:07.638525 31990 mini_ranger_kms.cc:227] Using postgres at 127.31.61.148:57397
2026-08-12 16:41:07,978 [I] DB FLAVOR :POSTGRES
2026-08-12 16:41:07,978 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 16:41:07,978 [I] Checking connection
2026-08-12 16:41:07,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 16:41:08,234 [I] connection success
2026-08-12 16:41:08,234 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 16:41:08,235 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 16:41:08,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 16:41:08,469 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 16:41:08,469 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 16:41:08,469 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 16:41:08,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.61.148:57397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 16:41:08,743 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 16:41:09.217228 31990 mini_ranger_kms.cc:310] Created kms service
I20260812 16:41:09.445295 31990 mini_ranger_kms.cc:326] Created kudu user
I20260812 16:41:09.493094 31990 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 16:41:09 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:09.909978 31990 mini_ranger_kms.cc:384] Added ranger policy
I20260812 16:41:09.910315 31990 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgT9LAJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgT9LAJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 16:41:09.910364 31990 mini_ranger_kms.cc:240] Using host: 127.31.61.148
I20260812 16:41:09.930408 31990 mini_ranger_kms.cc:276] Ranger KMS PID: 11992
I20260812 16:41:09.930528 31990 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.61.148:47473
Aug 12, 2026 4:41:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 4:41:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 4:41:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 4:41:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 4:41:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 4:41:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260812 16:41:12.101876 31990 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.553s user 0.049s sys 0.308s
I20260812 16:41:12.102133 31990 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 16:41:12.102293 31990 mini_ranger_kms.cc:401] 127.31.61.148:47473/kms/v1/keys
Aug 12 16:41:12 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786552872, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.61.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 16:41:12 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552872, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.61.148@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
Aug 12 16:41:13 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552872, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.61.148@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
Aug 12 16:41:15 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:41:15.492084 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43257
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:43257
--ranger_config_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:47473/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:15.846616 12044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:15.846923 12044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:15.847039 12044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:15.854554 12044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:15.854672 12044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:15.854746 12044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:15.854821 12044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:15.854883 12044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:15.865182 12044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.61.148:47473/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:43257
--ranger_config_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:15.866482 12044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:15.867866 12044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:15.879509 12053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:15.879545 12050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:15.879573 12051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:15.879851 12044 server_base.cc:1034] running on GCE node
I20260812 16:41:15.880792 12044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:15.882165 12044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:15.883355 12044 hybrid_clock.cc:648] HybridClock initialized: now 1786552875883317 us; error 30 us; skew 500 ppm
I20260812 16:41:15.883733 12044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:15 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552875, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:15.890719 12044 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:41:15.892280 12044 webserver.cc:460] Webserver started at http://127.31.61.190:33531/ using document root <none> and password file <none>
I20260812 16:41:15.892940 12044 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:15.893052 12044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:15.893352 12044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:41:15 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552875, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:15.944263 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "b22869ee243441959cdc182878a6e8ed"
format_stamp: "Formatted at 2026-08-12 16:41:15 on dist-test-slave-f7th"
server_key: "46cdad61ef4c014c59104be3bf06670d"
server_key_iv: "74ddd4a2f1501b777bdccaa9d26f79aa"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:15.945145 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "b22869ee243441959cdc182878a6e8ed"
format_stamp: "Formatted at 2026-08-12 16:41:15 on dist-test-slave-f7th"
server_key: "46cdad61ef4c014c59104be3bf06670d"
server_key_iv: "74ddd4a2f1501b777bdccaa9d26f79aa"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:15.951051 12044 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:41:15.972219 12061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:15.973990 12044 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:41:15.974205 12044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "b22869ee243441959cdc182878a6e8ed"
format_stamp: "Formatted at 2026-08-12 16:41:15 on dist-test-slave-f7th"
server_key: "46cdad61ef4c014c59104be3bf06670d"
server_key_iv: "74ddd4a2f1501b777bdccaa9d26f79aa"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:15.974443 12044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:15.991689 12044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:15.996062 12044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:15.996357 12044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:16.015293 12044 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:43257
I20260812 16:41:16.015324 12112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:43257 every 8 connection(s)
I20260812 16:41:16.016753 12044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:41:16.020604 12113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:16.021629 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 12044
I20260812 16:41:16.021806 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:41:16.022644 31990 external_mini_cluster.cc:1428] Setting key 6ce7874bc5662b66733a61c9952c4d27
I20260812 16:41:16.034699 12113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Bootstrap starting.
Aug 12 16:41:16 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:16.038056 12113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:16.039191 12113 log.cc:824] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:16.042227 12113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: No bootstrap required, opened a new log
I20260812 16:41:16.045687 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:16.024884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.31.61.190:43257)
0812 16:41:16.025734 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:16.025749 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:16.025803 (+ 54us) server_negotiation.cc:407] Connection header received
0812 16:41:16.027522 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:16.027558 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:16.028131 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:16.028550 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:16.029765 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:16.031789 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:16.033155 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:16.034145 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:16.034337 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:16.038379 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:16.038395 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:16.038413 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:16.038462 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:16.041620 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:16.042319 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:16.042328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:16.042337 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:16.042526 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:16.042956 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:16.042961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:16.042963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:16.043480 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:16.043802 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:16.044273 (+ 471us) server_negotiation.cc:299] Negotiation successful
0812 16:41:16.044686 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":384,"threads_started":1}
I20260812 16:41:16.050158 12113 raft_consensus.cc:348] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } }
I20260812 16:41:16.050522 12113 raft_consensus.cc:374] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:16.050642 12113 raft_consensus.cc:729] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22869ee243441959cdc182878a6e8ed, State: Initialized, Role: FOLLOWER
I20260812 16:41:16.051177 12113 consensus_queue.cc:260] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } }
I20260812 16:41:16.051395 12113 raft_consensus.cc:388] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:41:16.051518 12113 raft_consensus.cc:482] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:41:16.051699 12113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:16.053743 12113 raft_consensus.cc:504] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } }
I20260812 16:41:16.054206 12113 leader_election.cc:302] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b22869ee243441959cdc182878a6e8ed; no voters:
I20260812 16:41:16.054760 12113 leader_election.cc:288] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:41:16.054879 12118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:41:16.055537 12118 raft_consensus.cc:686] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 1 LEADER]: Becoming Leader. State: Replica: b22869ee243441959cdc182878a6e8ed, State: Running, Role: LEADER
I20260812 16:41:16.056078 12118 consensus_queue.cc:237] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } }
I20260812 16:41:16.056922 12113 sys_catalog.cc:564] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:41:16.058820 12119 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b22869ee243441959cdc182878a6e8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } } }
I20260812 16:41:16.059230 12119 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:16.058708 12120 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader b22869ee243441959cdc182878a6e8ed. Latest consensus state: current_term: 1 leader_uuid: "b22869ee243441959cdc182878a6e8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } } }
I20260812 16:41:16.059705 12120 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:16.060066 12126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:41:16.065557 12126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:41:16.066144 12113 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 16:41:16.084865 12126 catalog_manager.cc:1347] Generated new cluster ID: 0c717755798b4721bbc834c05cd40e57
I20260812 16:41:16.085040 12126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:41:16.101943 12126 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:41:16.103391 12126 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:41:16.112632 12126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Generated new TSK 0
I20260812 16:41:16.113667 12126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:41:16.831382 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:47473/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43257
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:41:17 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786552877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 16:41:17.247825 12159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:17.248260 12159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:17.248423 12159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:17.260191 12159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:17.260342 12159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:17.260591 12159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:41:17.276463 12159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.61.148:47473/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:17.278348 12159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:17.280283 12159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:17.298560 12170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:17.298780 12169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:17.299296 12172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:17.301208 12159 server_base.cc:1034] running on GCE node
I20260812 16:41:17.302052 12159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:17.303368 12159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:17.304626 12159 hybrid_clock.cc:648] HybridClock initialized: now 1786552877304538 us; error 64 us; skew 500 ppm
I20260812 16:41:17.305032 12159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:17 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:17.309284 12159 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:41:17.311271 12159 webserver.cc:460] Webserver started at http://127.31.61.129:45085/ using document root <none> and password file <none>
I20260812 16:41:17.312330 12159 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:17.312484 12159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:17.312846 12159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:41:17 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:17.333683 12159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "3802a0def17a4f7aa3d243566b4c6b85"
format_stamp: "Formatted at 2026-08-12 16:41:17 on dist-test-slave-f7th"
server_key: "8a727c4e023a82291efe98008177abe4"
server_key_iv: "3162bb098fac02036ee4dae25d035bb0"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:17.335021 12159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "3802a0def17a4f7aa3d243566b4c6b85"
format_stamp: "Formatted at 2026-08-12 16:41:17 on dist-test-slave-f7th"
server_key: "8a727c4e023a82291efe98008177abe4"
server_key_iv: "3162bb098fac02036ee4dae25d035bb0"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:17.342835 12159 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 16:41:17.380388 12179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:17.385069 12159 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.004s sys 0.008s
I20260812 16:41:17.385294 12159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "3802a0def17a4f7aa3d243566b4c6b85"
format_stamp: "Formatted at 2026-08-12 16:41:17 on dist-test-slave-f7th"
server_key: "8a727c4e023a82291efe98008177abe4"
server_key_iv: "3162bb098fac02036ee4dae25d035bb0"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:17.385586 12159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:17.431782 12159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:17.437363 12159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:17.437734 12159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:17.439087 12159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:17.441730 12159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:17.441857 12159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 16:41:17.442026 12159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:17.442114 12159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:17.486774 12159 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:35605
I20260812 16:41:17.492406 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 12159
I20260812 16:41:17.492612 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:41:17.493476 31990 external_mini_cluster.cc:1428] Setting key a05856642810a80334d4b22aab5d81ce
I20260812 16:41:17.486855 12292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:35605 every 8 connection(s)
I20260812 16:41:17.488400 12159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
Aug 12 16:41:17 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
I20260812 16:41:17.540743 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:17.512419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:36995 (local address 127.31.61.190:43257)
0812 16:41:17.517346 (+ 4927us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:17.517353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:17.519667 (+ 2314us) server_negotiation.cc:407] Connection header received
0812 16:41:17.521318 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:17.521325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:17.521440 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:17.521524 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:17.524006 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:17.525050 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:17.527343 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:17.527930 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:17.528032 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:17.532837 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:17.532846 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:17.532851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:17.532901 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:17.536375 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:17.537454 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:17.537459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:17.537461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:17.537599 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:17.538145 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:17.538150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:17.538154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:17.538640 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:17.538825 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:17.540114 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0812 16:41:17.540366 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4782,"thread_start_us":4670,"threads_started":1}
I20260812 16:41:17.543167 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:17.511621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43257 (local address 127.31.61.129:36995)
0812 16:41:17.517937 (+ 6316us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:17.517997 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:17.520770 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:17.522200 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:17.522213 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:17.522863 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:17.523755 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:17.523784 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:17.525276 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:17.525281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:17.527166 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:17.527183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:17.528409 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:17.528416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:17.528695 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:17.529498 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:17.529531 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:17.532494 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:17.536595 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:17.536606 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:17.536626 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:17.537275 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:17.537718 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:17.537722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:17.537725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:17.537989 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:17.538933 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:17.538943 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:17.539439 (+ 496us) client_negotiation.cc:769] Sending connection context
0812 16:41:17.540973 (+ 1534us) client_negotiation.cc:240] Negotiation successful
0812 16:41:17.541602 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6011,"thread_start_us":5763,"threads_started":1}
I20260812 16:41:17.545181 12293 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43257
I20260812 16:41:17.545534 12293 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:17.546393 12293 heartbeater.cc:511] Master 127.31.61.190:43257 requested a full tablet report, sending...
I20260812 16:41:17.549010 12078 ts_manager.cc:194] Registered new tserver with Master: 3802a0def17a4f7aa3d243566b4c6b85 (127.31.61.129:35605)
I20260812 16:41:17.550921 12078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:36995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:41:17.575655 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:47473/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43257
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:41:17 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
Aug 12 16:41:18 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
W20260812 16:41:18.059690 12299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:18.060248 12299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:18.060454 12299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:18.072082 12299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:18.072321 12299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:18.072584 12299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:41:18.088806 12299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.61.148:47473/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:18.090860 12299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:18.092690 12299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:18.132752 12306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:18.133291 12299 server_base.cc:1034] running on GCE node
W20260812 16:41:18.134202 12310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:18.135247 12307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:18.136247 12299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:18.137674 12299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:18.144310 12299 hybrid_clock.cc:648] HybridClock initialized: now 1786552878144189 us; error 103 us; skew 500 ppm
I20260812 16:41:18.144801 12299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:18 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:18.149647 12299 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:41:18.174283 12299 webserver.cc:460] Webserver started at http://127.31.61.130:42945/ using document root <none> and password file <none>
I20260812 16:41:18.175524 12299 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:18.175788 12299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:18.176216 12299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:41:18 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:18.208887 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "f8592ad0f4d1422492289daefcc5c76e"
format_stamp: "Formatted at 2026-08-12 16:41:18 on dist-test-slave-f7th"
server_key: "30af7f12a7235311625cbd5eea9efae3"
server_key_iv: "1b5298ad548333d802158f8bb94e6f2b"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:18.224522 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "f8592ad0f4d1422492289daefcc5c76e"
format_stamp: "Formatted at 2026-08-12 16:41:18 on dist-test-slave-f7th"
server_key: "30af7f12a7235311625cbd5eea9efae3"
server_key_iv: "1b5298ad548333d802158f8bb94e6f2b"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:18.243942 12299 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.005s sys 0.011s
I20260812 16:41:18.296458 12316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:18.301088 12299 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.018s sys 0.005s
I20260812 16:41:18.301363 12299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "f8592ad0f4d1422492289daefcc5c76e"
format_stamp: "Formatted at 2026-08-12 16:41:18 on dist-test-slave-f7th"
server_key: "30af7f12a7235311625cbd5eea9efae3"
server_key_iv: "1b5298ad548333d802158f8bb94e6f2b"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:18.301687 12299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:18.377068 12299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:18.383055 12299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:18.383483 12299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:18.386068 12299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:18.388142 12299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:18.388248 12299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:18.388418 12299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:18.388485 12299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:18.430718 12299 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:37511
I20260812 16:41:18.430742 12436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:37511 every 8 connection(s)
I20260812 16:41:18.432662 12299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:41:18.433066 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 12299
I20260812 16:41:18.433256 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:41:18.433800 31990 external_mini_cluster.cc:1428] Setting key 1a8555388d09793b48769774c0b4d0c9
Aug 12 16:41:18 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:18.465952 12440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:18.437557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43257 (local address 127.31.61.130:34843)
0812 16:41:18.441875 (+ 4318us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:18.441926 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:18.445239 (+ 3313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:18.447358 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:18.447370 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:18.448032 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:18.448805 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:18.448829 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:18.450385 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:18.450391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:18.452407 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:18.452431 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:18.453562 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:18.453571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:18.453852 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:18.454709 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:18.454742 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:18.457304 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:18.461274 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:18.461283 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:18.461302 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:18.461981 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:18.462436 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:18.462441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:18.462444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:18.462699 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:18.463345 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:18.463355 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:18.463806 (+ 451us) client_negotiation.cc:769] Sending connection context
0812 16:41:18.464089 (+ 283us) client_negotiation.cc:240] Negotiation successful
0812 16:41:18.464620 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4027,"thread_start_us":3775,"threads_started":1}
I20260812 16:41:18.467427 12439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:18.436793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:34843 (local address 127.31.61.190:43257)
0812 16:41:18.440339 (+ 3546us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:18.440347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:18.442213 (+ 1866us) server_negotiation.cc:407] Connection header received
0812 16:41:18.446711 (+ 4498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:18.446717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:18.446821 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:18.446923 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:18.449016 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:18.450224 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:18.452634 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:18.453251 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:18.454256 (+ 1005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:18.457576 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:18.457583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:18.457588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:18.457629 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:18.461116 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:18.462152 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:18.462158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:18.462162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:18.462314 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:18.462841 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:18.462846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:18.462849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:18.463201 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:18.466678 (+ 3477us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:18.466834 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 16:41:18.467075 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3399,"thread_start_us":3274,"threads_started":1}
I20260812 16:41:18.468911 12437 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43257
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
I20260812 16:41:18.469228 12437 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:18.470075 12437 heartbeater.cc:511] Master 127.31.61.190:43257 requested a full tablet report, sending...
I20260812 16:41:18.471840 12078 ts_manager.cc:194] Registered new tserver with Master: f8592ad0f4d1422492289daefcc5c76e (127.31.61.130:37511)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
I20260812 16:41:18.472941 12078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:34843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:41:18.507773 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:47473/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43257
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:41:18.555078 12293 heartbeater.cc:503] Master 127.31.61.190:43257 was elected leader, sending a full tablet report...
W20260812 16:41:18.905262 12443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:18.905548 12443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:18.905681 12443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:18.913199 12443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:18.913313 12443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:18.913471 12443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:41:18.923187 12443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.61.148:47473/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:18.924499 12443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:18.925748 12443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:18.938319 12448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:18.938356 12449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:18.938234 12451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:18.939345 12443 server_base.cc:1034] running on GCE node
I20260812 16:41:18.940040 12443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:18.941004 12443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:18.942237 12443 hybrid_clock.cc:648] HybridClock initialized: now 1786552878942183 us; error 36 us; skew 500 ppm
I20260812 16:41:18.942611 12443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:18 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:18.945748 12443 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:41:18.947204 12443 webserver.cc:460] Webserver started at http://127.31.61.131:33031/ using document root <none> and password file <none>
I20260812 16:41:18.947945 12443 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:18.948062 12443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:18.948372 12443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 16:41:18 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:18.965952 12443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "2cc5f121322e4c5ca7ded68729141157"
format_stamp: "Formatted at 2026-08-12 16:41:18 on dist-test-slave-f7th"
server_key: "929602374f78559ce9b58042b4400591"
server_key_iv: "a4733b31163f9b48bbcd76837bf80280"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:18.966856 12443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "2cc5f121322e4c5ca7ded68729141157"
format_stamp: "Formatted at 2026-08-12 16:41:18 on dist-test-slave-f7th"
server_key: "929602374f78559ce9b58042b4400591"
server_key_iv: "a4733b31163f9b48bbcd76837bf80280"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:18.972692 12443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 16:41:18.989790 12458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:18.991535 12443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 16:41:18.991791 12443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "2cc5f121322e4c5ca7ded68729141157"
format_stamp: "Formatted at 2026-08-12 16:41:18 on dist-test-slave-f7th"
server_key: "929602374f78559ce9b58042b4400591"
server_key_iv: "a4733b31163f9b48bbcd76837bf80280"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:18.992034 12443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:19.020581 12443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:19.024513 12443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:19.024811 12443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:19.025784 12443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:19.027176 12443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:19.027252 12443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:19.027374 12443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:19.027432 12443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:19.059769 12443 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:34763
I20260812 16:41:19.059788 12571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:34763 every 8 connection(s)
I20260812 16:41:19.061185 12443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:41:19.063995 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 12443
I20260812 16:41:19.064172 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:41:19.064960 31990 external_mini_cluster.cc:1428] Setting key b8bc281d65527fb6c39faa689e6a2fbb
Aug 12 16:41:19 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:19.082454 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:19.063768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:46545 (local address 127.31.61.190:43257)
0812 16:41:19.064212 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:19.064217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:19.065439 (+ 1222us) server_negotiation.cc:407] Connection header received
0812 16:41:19.067677 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:19.067683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:19.067798 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:19.067888 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:19.069565 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.070313 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.071930 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.072403 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.072515 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:19.076276 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:19.076281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:19.076284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:19.076329 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:19.079252 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:19.080100 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:19.080105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:19.080109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:19.080289 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:19.080831 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:19.080835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:19.080837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:19.081137 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:19.081257 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:19.081943 (+ 686us) server_negotiation.cc:299] Negotiation successful
0812 16:41:19.082120 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":241,"threads_started":1}
I20260812 16:41:19.083469 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:19.064463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43257 (local address 127.31.61.131:46545)
0812 16:41:19.065282 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:19.065319 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:19.067410 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:19.068180 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:19.068188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:19.068779 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:19.069395 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:19.069412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.070473 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.070477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:19.071792 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:19.071804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.072629 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.072637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:19.072960 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:19.073629 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:19.073656 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:19.076044 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:19.079433 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:19.079441 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:19.079456 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:19.079977 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:19.080402 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:19.080405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:19.080408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:19.080710 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:19.081243 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:19.081250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:19.081570 (+ 320us) client_negotiation.cc:769] Sending connection context
0812 16:41:19.081816 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 16:41:19.082243 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":368,"threads_started":1}
I20260812 16:41:19.084910 12572 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43257
I20260812 16:41:19.085235 12572 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:19.086088 12572 heartbeater.cc:511] Master 127.31.61.190:43257 requested a full tablet report, sending...
I20260812 16:41:19.087695 12078 ts_manager.cc:194] Registered new tserver with Master: 2cc5f121322e4c5ca7ded68729141157 (127.31.61.131:34763)
I20260812 16:41:19.088619 12078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:46545
I20260812 16:41:19.093143 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:41:19.094821 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 12159
Aug 12 16:41:19 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:41:19.114418 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:19.095968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.31.61.129:35605)
0812 16:41:19.096746 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:19.096753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:19.096778 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:41:19.096831 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:19.096837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:19.097051 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:19.097170 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:19.098822 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.099950 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.101693 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.102298 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.102402 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:19.107124 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:19.107138 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:19.107172 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:19.107219 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:19.111040 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:19.111825 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:19.111835 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:19.111838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:19.112136 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:19.112514 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:19.112517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:19.112519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:19.112883 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:19.113143 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:19.113552 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 16:41:19.113805 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":423,"threads_started":1}
I20260812 16:41:19.115921 12288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:19.203387 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 12159
I20260812 16:41:19.219964 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 12299
Aug 12 16:41:19 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:41:19.237130 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:19.220884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.31.61.130:37511)
0812 16:41:19.221386 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:19.221393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:19.221419 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:41:19.221490 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:19.221496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:19.221693 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:19.221833 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:19.223293 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.224335 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.225807 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.226442 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.226694 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:19.231138 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:19.231152 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:19.231169 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:19.231213 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:19.234057 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:19.234824 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:19.234831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:19.234833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:19.234981 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:19.235368 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:19.235374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:19.235377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:19.235869 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:19.236133 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:19.236513 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 16:41:19.236785 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":263,"threads_started":1}
I20260812 16:41:19.238380 12432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:19.320680 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 12299
I20260812 16:41:19.336683 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 12443
Aug 12 16:41:19 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:41:19.353715 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:19.337669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.31.61.131:34763)
0812 16:41:19.338258 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:19.338265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:19.338296 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:41:19.338364 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:19.338371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:19.338594 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:19.338722 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:19.340213 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.341299 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.342705 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:19.343269 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:19.343381 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:19.347936 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:19.347950 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:19.347965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:19.348006 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:19.350729 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:19.351484 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:19.351511 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:19.351514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:19.351697 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:19.352083 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:19.352087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:19.352089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:19.352467 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:19.352737 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:19.353172 (+ 435us) server_negotiation.cc:299] Negotiation successful
0812 16:41:19.353382 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":292,"threads_started":1}
I20260812 16:41:19.355114 12567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:19.437776 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 12443
I20260812 16:41:19.452848 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 12044
I20260812 16:41:19.454236 12108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:19.559753 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 12044
I20260812 16:41:19.582233 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43257
--webserver_interface=127.31.61.190
--webserver_port=33531
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:43257
--ranger_config_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:47473/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.61.148:47473/kms with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:19.907428 12584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:19.907773 12584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:19.907912 12584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:19.915170 12584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:19.915294 12584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:19.915366 12584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:19.915441 12584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:19.915505 12584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:19.925813 12584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.61.148:47473/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:43257
--ranger_config_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=33531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:19.927129 12584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:19.928463 12584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:19.939905 12590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:19.939924 12593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:19.940016 12591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:19.940639 12584 server_base.cc:1034] running on GCE node
I20260812 16:41:19.941386 12584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:19.942739 12584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:19.943926 12584 hybrid_clock.cc:648] HybridClock initialized: now 1786552879943870 us; error 45 us; skew 500 ppm
I20260812 16:41:19.944279 12584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:19 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:19.947361 12584 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:41:19.949029 12584 webserver.cc:460] Webserver started at http://127.31.61.190:33531/ using document root <none> and password file <none>
I20260812 16:41:19.949780 12584 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:19.949911 12584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 16:41:19 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:20.015641 12584 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 16:41:20.018702 12601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:20.020105 12584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:41:20.020283 12584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "b22869ee243441959cdc182878a6e8ed"
format_stamp: "Formatted at 2026-08-12 16:41:15 on dist-test-slave-f7th"
server_key: "46cdad61ef4c014c59104be3bf06670d"
server_key_iv: "74ddd4a2f1501b777bdccaa9d26f79aa"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:20.021277 12584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:20.042003 12584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:20.045703 12584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:20.045974 12584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:20.063465 12584 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:43257
I20260812 16:41:20.063508 12654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:43257 every 8 connection(s)
I20260812 16:41:20.065039 12584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:41:20.066963 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 12584
I20260812 16:41:20.068064 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:47473/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:35605
--local_ip_for_outbound_sockets=127.31.61.129
--tserver_master_addrs=127.31.61.190:43257
--webserver_port=45085
--webserver_interface=127.31.61.129
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.61.148:47473/kms with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:41:20.074164 12655 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:41:20.078250 12655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Bootstrap starting.
I20260812 16:41:20.097013 12655 log.cc:824] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:20.106436 12655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:41:20.107005 12655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Bootstrap complete.
I20260812 16:41:20.115918 12655 raft_consensus.cc:348] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } }
I20260812 16:41:20.116845 12655 raft_consensus.cc:729] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22869ee243441959cdc182878a6e8ed, State: Initialized, Role: FOLLOWER
I20260812 16:41:20.117486 12655 consensus_queue.cc:260] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } }
I20260812 16:41:20.117698 12655 raft_consensus.cc:388] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:41:20.117813 12655 raft_consensus.cc:482] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:41:20.117973 12655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:41:20.122205 12655 raft_consensus.cc:504] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } }
I20260812 16:41:20.122792 12655 leader_election.cc:302] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b22869ee243441959cdc182878a6e8ed; no voters:
I20260812 16:41:20.123404 12655 leader_election.cc:288] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 16:41:20.123620 12659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:41:20.124986 12659 raft_consensus.cc:686] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [term 2 LEADER]: Becoming Leader. State: Replica: b22869ee243441959cdc182878a6e8ed, State: Running, Role: LEADER
I20260812 16:41:20.125758 12659 consensus_queue.cc:237] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } }
I20260812 16:41:20.126461 12655 sys_catalog.cc:564] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:41:20.128876 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader b22869ee243441959cdc182878a6e8ed. Latest consensus state: current_term: 2 leader_uuid: "b22869ee243441959cdc182878a6e8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } } }
I20260812 16:41:20.129297 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:20.130015 12660 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b22869ee243441959cdc182878a6e8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22869ee243441959cdc182878a6e8ed" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43257 } } }
I20260812 16:41:20.130434 12660 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:20.138336 12664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:41:20.144567 12664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:41:20.146263 12664 catalog_manager.cc:1259] Loaded cluster ID: 0c717755798b4721bbc834c05cd40e57
I20260812 16:41:20.146445 12664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:41:20.150386 12664 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:41:20.152307 12664 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Loaded TSK: 0
I20260812 16:41:20.154234 12664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:41:20.158951 12655 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260812 16:41:20.483223 12657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:20.483877 12657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:20.484100 12657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:20.496387 12657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:20.496632 12657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:20.496924 12657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:41:20.512302 12657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.61.148:47473/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:35605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=45085
--tserver_master_addrs=127.31.61.190:43257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:20.514242 12657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:20.515944 12657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:20.536980 12692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:20.548350 12693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:20.555083 12695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:20.557706 12657 server_base.cc:1034] running on GCE node
I20260812 16:41:20.558475 12657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:20.559862 12657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:20.561199 12657 hybrid_clock.cc:648] HybridClock initialized: now 1786552880561123 us; error 60 us; skew 500 ppm
I20260812 16:41:20.561688 12657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:20 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:20.567510 12657 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:41:20.583107 12657 webserver.cc:460] Webserver started at http://127.31.61.129:45085/ using document root <none> and password file <none>
I20260812 16:41:20.583961 12657 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:20.584129 12657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 16:41:20 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:20.623670 12657 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.005s sys 0.000s
I20260812 16:41:20.627700 12703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:20.629442 12657 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:41:20.629639 12657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "3802a0def17a4f7aa3d243566b4c6b85"
format_stamp: "Formatted at 2026-08-12 16:41:17 on dist-test-slave-f7th"
server_key: "8a727c4e023a82291efe98008177abe4"
server_key_iv: "3162bb098fac02036ee4dae25d035bb0"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:20.631031 12657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:20.674355 12657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:20.679482 12657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:20.679806 12657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:20.680974 12657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:20.682546 12657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:20.682648 12657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:20.682763 12657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:20.682847 12657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:20.786918 12657 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:35605
I20260812 16:41:20.786975 12817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:35605 every 8 connection(s)
I20260812 16:41:20.788400 12657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:41:20.791688 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 12657
I20260812 16:41:20.792553 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:47473/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:37511
--local_ip_for_outbound_sockets=127.31.61.130
--tserver_master_addrs=127.31.61.190:43257
--webserver_port=42945
--webserver_interface=127.31.61.130
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.61.148:47473/kms with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:41:20 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:20.834471 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:20.792036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43257 (local address 127.31.61.129:46719)
0812 16:41:20.803888 (+ 11852us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:20.803952 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:20.806590 (+ 2638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:20.810492 (+ 3902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:20.810503 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:20.811181 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:20.812098 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:20.812130 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:20.814658 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:20.814664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:20.816888 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:20.816909 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:20.818890 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:20.818901 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:20.819224 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:20.820005 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:20.820060 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:20.823052 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:20.828376 (+ 5324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:20.828387 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:20.828406 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:20.829136 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:20.829779 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:20.829784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:20.829788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:20.830115 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:20.831406 (+ 1291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:20.831417 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:20.831963 (+ 546us) client_negotiation.cc:769] Sending connection context
0812 16:41:20.832299 (+ 336us) client_negotiation.cc:240] Negotiation successful
0812 16:41:20.832921 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11455,"thread_start_us":14960,"threads_started":1}
I20260812 16:41:20.837715 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:20.791902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:46719 (local address 127.31.61.190:43257)
0812 16:41:20.803146 (+ 11244us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:20.803172 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:20.807997 (+ 4825us) server_negotiation.cc:407] Connection header received
0812 16:41:20.808556 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:20.808581 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:20.809133 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:20.809584 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:20.812383 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:20.814424 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:20.817102 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:20.818236 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:20.818435 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:20.823459 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:20.823472 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:20.823496 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:20.823724 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:20.828116 (+ 4392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:20.829347 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:20.829357 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:20.829365 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:20.829610 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:20.830270 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:20.830276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:20.830280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:20.831007 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:20.831303 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:20.835337 (+ 4034us) server_negotiation.cc:299] Negotiation successful
0812 16:41:20.836022 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10894,"thread_start_us":10596,"threads_started":1}
I20260812 16:41:20.842079 12818 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43257
I20260812 16:41:20.842448 12818 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:20.843349 12818 heartbeater.cc:511] Master 127.31.61.190:43257 requested a full tablet report, sending...
W20260812 16:41:20.846709 12818 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:43257 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 16:41:20.846949 12818 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:20.847237 12818 heartbeater.cc:511] Master 127.31.61.190:43257 requested a full tablet report, sending...
I20260812 16:41:20.848942 12818 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:20.849318 12818 heartbeater.cc:511] Master 127.31.61.190:43257 requested a full tablet report, sending...
W20260812 16:41:20.850508 12818 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:41:21.196755 12823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:21.197055 12823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:21.197229 12823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:21.204973 12823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:21.205103 12823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:21.205267 12823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:41:21.215468 12823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.61.148:47473/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:37511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=42945
--tserver_master_addrs=127.31.61.190:43257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:21.216859 12823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:21.218259 12823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:21.234090 12836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:21.234189 12835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:21.235357 12838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:21.235819 12823 server_base.cc:1034] running on GCE node
I20260812 16:41:21.236472 12823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:21.237573 12823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:21.238801 12823 hybrid_clock.cc:648] HybridClock initialized: now 1786552881238740 us; error 50 us; skew 500 ppm
I20260812 16:41:21.239113 12823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:21 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552881, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:21.250504 12823 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:41:21.257493 12823 webserver.cc:460] Webserver started at http://127.31.61.130:42945/ using document root <none> and password file <none>
I20260812 16:41:21.258352 12823 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:21.258508 12823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 16:41:21 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552881, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:21.280546 12823 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:41:21.283386 12846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:21.284823 12823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:41:21.285000 12823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "f8592ad0f4d1422492289daefcc5c76e"
format_stamp: "Formatted at 2026-08-12 16:41:18 on dist-test-slave-f7th"
server_key: "30af7f12a7235311625cbd5eea9efae3"
server_key_iv: "1b5298ad548333d802158f8bb94e6f2b"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:21.286018 12823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:21.317554 12823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:21.322724 12823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:21.323068 12823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:21.324375 12823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:21.326216 12823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:21.326332 12823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:21.326447 12823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:21.326524 12823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:21.444491 12823 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:37511
I20260812 16:41:21.444655 12964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:37511 every 8 connection(s)
I20260812 16:41:21.445978 12823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:41:21.453049 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 12823
I20260812 16:41:21.453919 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.61.148:47473/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:34763
--local_ip_for_outbound_sockets=127.31.61.131
--tserver_master_addrs=127.31.61.190:43257
--webserver_port=33031
--webserver_interface=127.31.61.131
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.61.148:47473/kms with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:41:21 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786552881, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 16:41:21 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552881, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:21.490950 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:21.449342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:46405 (local address 127.31.61.190:43257)
0812 16:41:21.464639 (+ 15297us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:21.464646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:21.469650 (+ 5004us) server_negotiation.cc:407] Connection header received
0812 16:41:21.472815 (+ 3165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:21.472821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:21.472917 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:21.473041 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:21.475217 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:21.476366 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:21.478672 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:21.479257 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:21.479344 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:21.483469 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:21.483475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:21.483479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:21.483522 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:21.487049 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:21.488008 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:21.488014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:21.488017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:21.488176 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:21.488761 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:21.488767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:21.488770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:21.489167 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:21.489318 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:21.490330 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0812 16:41:21.490565 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15134,"thread_start_us":15302,"threads_started":1}
I20260812 16:41:21.493487 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:21.451043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43257 (local address 127.31.61.130:46405)
0812 16:41:21.469427 (+ 18384us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:21.469503 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:21.472376 (+ 2873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:21.473513 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:21.473525 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:21.474138 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:21.474918 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:21.474944 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:21.476736 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:21.476741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:21.478512 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:21.478535 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:21.479725 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:21.479732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:21.480017 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:21.480759 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:21.480787 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:21.483212 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:21.487226 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:21.487236 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:21.487254 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:21.487852 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:21.488334 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:21.488338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:21.488342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:21.488620 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:21.489416 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:21.489423 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:21.489874 (+ 451us) client_negotiation.cc:769] Sending connection context
0812 16:41:21.491514 (+ 1640us) client_negotiation.cc:240] Negotiation successful
0812 16:41:21.492124 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17940,"thread_start_us":24417,"threads_started":1}
I20260812 16:41:21.495388 12965 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43257
I20260812 16:41:21.495728 12965 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:21.496563 12965 heartbeater.cc:511] Master 127.31.61.190:43257 requested a full tablet report, sending...
I20260812 16:41:21.498416 12618 ts_manager.cc:194] Registered new tserver with Master: f8592ad0f4d1422492289daefcc5c76e (127.31.61.130:37511)
I20260812 16:41:21.500196 12618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:46405
I20260812 16:41:21.850683 12818 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:21.851213 12818 heartbeater.cc:511] Master 127.31.61.190:43257 requested a full tablet report, sending...
W20260812 16:41:21.851033 12969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:21.851645 12969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:21.851861 12969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 16:41:21.852579 12618 ts_manager.cc:194] Registered new tserver with Master: 3802a0def17a4f7aa3d243566b4c6b85 (127.31.61.129:35605)
I20260812 16:41:21.853734 12618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:46719
W20260812 16:41:21.862914 12969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:21.863073 12969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:21.863265 12969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:41:21.879312 12969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.61.148:47473/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:34763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=33031
--tserver_master_addrs=127.31.61.190:43257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:21.881022 12969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:21.882705 12969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:21.899729 12975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:21.900688 12976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:21.912424 12969 server_base.cc:1034] running on GCE node
W20260812 16:41:21.912732 12978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:21.913648 12969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:21.915315 12969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:21.916574 12969 hybrid_clock.cc:648] HybridClock initialized: now 1786552881916494 us; error 64 us; skew 500 ppm
I20260812 16:41:21.916935 12969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:21 dist-test-slave-f7th krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552881, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:21.920322 12969 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
Aug 12 16:41:21 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552881, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:21.928823 12969 webserver.cc:460] Webserver started at http://127.31.61.131:33031/ using document root <none> and password file <none>
I20260812 16:41:21.929595 12969 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:21.929741 12969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 16:41:21 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552881, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:21.957864 12969 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:41:21.965188 12986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:21.967276 12969 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260812 16:41:21.967558 12969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "2cc5f121322e4c5ca7ded68729141157"
format_stamp: "Formatted at 2026-08-12 16:41:18 on dist-test-slave-f7th"
server_key: "929602374f78559ce9b58042b4400591"
server_key_iv: "a4733b31163f9b48bbcd76837bf80280"
server_key_version: "kuduclusterkey@0"
I20260812 16:41:21.970763 12969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:22.059643 12969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:22.065066 12969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:22.065382 12969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:22.073935 12969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:22.075928 12969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:22.076048 12969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:22.076165 12969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:22.076237 12969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:22.121776 12969 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:34763
I20260812 16:41:22.121881 13099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:34763 every 8 connection(s)
I20260812 16:41:22.123627 12969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:41:22.128558 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 12969
Aug 12 16:41:22 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552881, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:22.158490 13105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.130480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.31.61.190:43257)
0812 16:41:22.138943 (+ 8463us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.138948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.139794 (+ 846us) server_negotiation.cc:407] Connection header received
0812 16:41:22.139883 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.139888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.139998 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.140105 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:22.142365 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.143451 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.145434 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.146092 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.146211 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.149807 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:22.149814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:22.149819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:22.149861 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:22.154014 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.154927 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.154932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.154936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.155088 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.155549 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.155555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.155558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.155945 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:22.156228 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.157775 (+ 1547us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.158040 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8368,"thread_start_us":8234,"threads_started":1}
I20260812 16:41:22.164317 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.129358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:36325 (local address 127.31.61.190:43257)
0812 16:41:22.138600 (+ 9242us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.138607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.142289 (+ 3682us) server_negotiation.cc:407] Connection header received
0812 16:41:22.144783 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.144789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.144873 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.144978 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:22.147373 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.148624 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.151173 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.151975 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.152078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.156770 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:22.156777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:22.156781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:22.156817 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:22.160340 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.161371 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.161376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.161379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.161542 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.162134 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.162140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.162143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.162492 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:22.162631 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.163699 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.163931 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9092,"thread_start_us":8971,"threads_started":1}
I20260812 16:41:22.166615 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.127589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43257 (local address 127.31.61.131:36325)
0812 16:41:22.140743 (+ 13154us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.140809 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:22.143383 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:22.145490 (+ 2107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:22.145505 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:22.146211 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:22.147111 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.147140 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.148844 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.148849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.150986 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.151011 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.152600 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.152610 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.152923 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.153615 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:22.153654 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:22.156441 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:22.160523 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:22.160548 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:22.160568 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:22.161221 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:22.161713 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:22.161718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:22.161722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:22.162001 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:22.162749 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:22.162759 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:22.163224 (+ 465us) client_negotiation.cc:769] Sending connection context
0812 16:41:22.164614 (+ 1390us) client_negotiation.cc:240] Negotiation successful
0812 16:41:22.165210 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12730,"thread_start_us":16123,"threads_started":1}
I20260812 16:41:22.168295 13100 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43257
I20260812 16:41:22.168648 13100 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:22.169530 13100 heartbeater.cc:511] Master 127.31.61.190:43257 requested a full tablet report, sending...
I20260812 16:41:22.171293 12618 ts_manager.cc:194] Registered new tserver with Master: 2cc5f121322e4c5ca7ded68729141157 (127.31.61.131:34763)
I20260812 16:41:22.172516 12618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:36325
I20260812 16:41:22.174569 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:41:22.199481 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.182251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.31.61.190:43257)
0812 16:41:22.182457 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.182463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.183014 (+ 551us) server_negotiation.cc:407] Connection header received
0812 16:41:22.183173 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.183179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.183275 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.183400 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:22.185493 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.186641 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.188520 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.189198 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.189357 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.192503 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:22.192510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:22.192514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:22.192560 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:22.196124 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.197026 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.197033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.197036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.197195 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.197652 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.197657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.197660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.198049 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:22.198298 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.198664 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.198908 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:41:22.205803 12618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:41:22.217837 12618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 12 16:41:22 dist-test-slave-f7th krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552881, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for HTTP/127.31.61.148@KRBTEST.COM
I20260812 16:41:22.275250 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.252267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:34763 (local address 127.0.0.1:44592)
0812 16:41:22.265010 (+ 12743us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.265065 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:22.265264 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:22.266999 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:22.267006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:22.267062 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:22.267486 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.267501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.269597 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.269603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.272183 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.272206 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.274313 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.274319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.274518 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.274578 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 16:41:22.274702 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 16:41:22.274891 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12429,"thread_start_us":12269,"threads_started":1}
I20260812 16:41:22.276156 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.252517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.31.61.131:34763)
0812 16:41:22.265856 (+ 13339us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.265865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.265898 (+ 33us) server_negotiation.cc:407] Connection header received
0812 16:41:22.265993 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.266000 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.266274 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.266458 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:22.267782 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.269419 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.272393 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.273787 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.273921 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.274163 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.275664 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.275846 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13164,"thread_start_us":14202,"threads_started":1}
I20260812 16:41:22.280685 13035 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1d8dfd98a647739e278d448fa851af (DEFAULT_TABLE table=test-table [id=0d6edf2c438d4183a38d0f3172489e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:22.282409 13035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1d8dfd98a647739e278d448fa851af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:22.321776 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.257382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.0.0.1:56030)
0812 16:41:22.287937 (+ 30555us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.287961 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:22.288106 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:22.313311 (+ 25205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:22.313317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:22.313359 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:22.313575 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.313601 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.315697 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.315702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.317498 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.317514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.321003 (+ 3489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.321014 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.321204 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.321245 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 16:41:22.321319 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:41:22.321461 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30415,"thread_start_us":63757,"threads_started":1}
I20260812 16:41:22.323107 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.258055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.31.61.129:35605)
0812 16:41:22.312005 (+ 53950us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.312544 (+ 539us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.312589 (+ 45us) server_negotiation.cc:407] Connection header received
0812 16:41:22.312716 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.312725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.313038 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.313290 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:22.313810 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.315404 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.317788 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.319198 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.322291 (+ 3093us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.322526 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.322627 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.322797 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49329,"thread_start_us":50606,"threads_started":1}
I20260812 16:41:22.323459 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.271965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:37511 (local address 127.0.0.1:60402)
0812 16:41:22.310172 (+ 38207us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.312049 (+ 1877us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:22.313664 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:22.314919 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:22.314924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:22.314950 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:22.315266 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.315277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.317081 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.317085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.319089 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.319107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.322682 (+ 3575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.322689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.322863 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.322898 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:41:22.322981 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 16:41:22.323114 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38048,"thread_start_us":39406,"threads_started":1}
I20260812 16:41:22.324647 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.272648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.31.61.130:37511)
0812 16:41:22.308639 (+ 35991us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.308648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.313791 (+ 5143us) server_negotiation.cc:407] Connection header received
0812 16:41:22.313905 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.313913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.314123 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.314305 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:22.315450 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.316896 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.319273 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.320734 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.320865 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.321120 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.324043 (+ 2923us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.324218 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35782,"thread_start_us":36292,"threads_started":1}
I20260812 16:41:22.328687 12752 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1d8dfd98a647739e278d448fa851af (DEFAULT_TABLE table=test-table [id=0d6edf2c438d4183a38d0f3172489e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:22.331028 12752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1d8dfd98a647739e278d448fa851af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:22.335903 12897 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1d8dfd98a647739e278d448fa851af (DEFAULT_TABLE table=test-table [id=0d6edf2c438d4183a38d0f3172489e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:22.338016 12897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1d8dfd98a647739e278d448fa851af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:22.354884 13122 tablet_bootstrap.cc:492] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: Bootstrap starting.
I20260812 16:41:22.365525 13122 tablet_bootstrap.cc:654] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:22.366988 13122 log.cc:824] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:22.380846 13124 tablet_bootstrap.cc:492] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: Bootstrap starting.
I20260812 16:41:22.388067 13124 tablet_bootstrap.cc:654] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:22.396479 13125 tablet_bootstrap.cc:492] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: Bootstrap starting.
I20260812 16:41:22.397997 13122 tablet_bootstrap.cc:492] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: No bootstrap required, opened a new log
I20260812 16:41:22.398312 13122 ts_tablet_manager.cc:1397] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.004s
I20260812 16:41:22.400688 13125 tablet_bootstrap.cc:654] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:22.402123 13125 log.cc:824] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:22.404151 13124 log.cc:824] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:22.406874 13122 raft_consensus.cc:348] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } }
I20260812 16:41:22.407524 13122 raft_consensus.cc:374] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:22.407697 13122 raft_consensus.cc:729] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc5f121322e4c5ca7ded68729141157, State: Initialized, Role: FOLLOWER
I20260812 16:41:22.408595 13122 consensus_queue.cc:260] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } }
I20260812 16:41:22.413762 13125 tablet_bootstrap.cc:492] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: No bootstrap required, opened a new log
I20260812 16:41:22.414130 13125 ts_tablet_manager.cc:1397] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20260812 16:41:22.418260 13124 tablet_bootstrap.cc:492] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: No bootstrap required, opened a new log
I20260812 16:41:22.418565 13124 ts_tablet_manager.cc:1397] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.004s
I20260812 16:41:22.426133 13125 raft_consensus.cc:348] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } }
I20260812 16:41:22.426801 13125 raft_consensus.cc:374] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:22.426949 13125 raft_consensus.cc:729] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8592ad0f4d1422492289daefcc5c76e, State: Initialized, Role: FOLLOWER
I20260812 16:41:22.427881 13125 consensus_queue.cc:260] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } }
I20260812 16:41:22.432605 13124 raft_consensus.cc:348] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } }
I20260812 16:41:22.433110 13124 raft_consensus.cc:374] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:22.433218 13124 raft_consensus.cc:729] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3802a0def17a4f7aa3d243566b4c6b85, State: Initialized, Role: FOLLOWER
I20260812 16:41:22.435472 12965 heartbeater.cc:503] Master 127.31.61.190:43257 was elected leader, sending a full tablet report...
I20260812 16:41:22.436648 13125 ts_tablet_manager.cc:1428] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: Time spent starting tablet: real 0.022s user 0.018s sys 0.004s
I20260812 16:41:22.449885 13100 heartbeater.cc:503] Master 127.31.61.190:43257 was elected leader, sending a full tablet report...
I20260812 16:41:22.450618 13122 ts_tablet_manager.cc:1428] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: Time spent starting tablet: real 0.052s user 0.032s sys 0.014s
I20260812 16:41:22.451458 13124 consensus_queue.cc:260] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } }
I20260812 16:41:22.452931 12818 heartbeater.cc:503] Master 127.31.61.190:43257 was elected leader, sending a full tablet report...
I20260812 16:41:22.453639 13124 ts_tablet_manager.cc:1428] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: Time spent starting tablet: real 0.035s user 0.015s sys 0.000s
W20260812 16:41:22.474368 12966 tablet.cc:2369] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:41:22.546583 12819 tablet.cc:2369] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:41:22.559051 13128 raft_consensus.cc:482] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:22.559348 13128 raft_consensus.cc:504] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } }
I20260812 16:41:22.561146 13128 leader_election.cc:288] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3802a0def17a4f7aa3d243566b4c6b85 (127.31.61.129:35605), f8592ad0f4d1422492289daefcc5c76e (127.31.61.130:37511)
I20260812 16:41:22.568121 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.561368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.31.61.131:39853)
0812 16:41:22.561651 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.561670 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:22.561802 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:22.562388 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:22.562394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:22.562423 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:22.562676 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.562689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.564251 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.564257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.565794 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.565807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.567310 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.567330 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.567519 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.567556 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:41:22.567699 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 16:41:22.567846 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260812 16:41:22.568463 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.561736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39853 (local address 127.31.61.129:35605)
0812 16:41:22.561936 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.561941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.561960 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:41:22.562032 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.562037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.562112 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.562202 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:22.562860 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.564102 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.566004 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.567102 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.567269 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.567371 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.567742 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.567896 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:41:22.568985 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.562098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:38791 (local address 127.31.61.130:37511)
0812 16:41:22.562294 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.562299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.562799 (+ 500us) server_negotiation.cc:407] Connection header received
0812 16:41:22.563013 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.563016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.563087 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.563160 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:22.563770 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.564583 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.566687 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.567678 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.567821 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.567886 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.568659 (+ 773us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.568796 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:41:22.569041 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.561620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:37511 (local address 127.31.61.131:38791)
0812 16:41:22.562715 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.562731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:22.562882 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:22.563192 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:22.563196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:22.563219 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:22.563620 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.563633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.564760 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.564764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.566362 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:22.566378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.567880 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.567887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:22.568412 (+ 525us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.568450 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:41:22.568559 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 16:41:22.568705 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":759,"threads_started":1}
I20260812 16:41:22.569592 12773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1d8dfd98a647739e278d448fa851af" candidate_uuid: "2cc5f121322e4c5ca7ded68729141157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3802a0def17a4f7aa3d243566b4c6b85" is_pre_election: true
I20260812 16:41:22.569932 12919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1d8dfd98a647739e278d448fa851af" candidate_uuid: "2cc5f121322e4c5ca7ded68729141157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8592ad0f4d1422492289daefcc5c76e" is_pre_election: true
I20260812 16:41:22.570267 12773 raft_consensus.cc:2454] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc5f121322e4c5ca7ded68729141157 in term 0.
I20260812 16:41:22.570462 12919 raft_consensus.cc:2454] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc5f121322e4c5ca7ded68729141157 in term 0.
I20260812 16:41:22.571002 12987 leader_election.cc:302] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc5f121322e4c5ca7ded68729141157, 3802a0def17a4f7aa3d243566b4c6b85; no voters:
I20260812 16:41:22.571451 13128 raft_consensus.cc:2781] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:41:22.571641 13128 raft_consensus.cc:482] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:41:22.571751 13128 raft_consensus.cc:3037] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:22.573894 13128 raft_consensus.cc:504] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } }
I20260812 16:41:22.574482 13128 leader_election.cc:288] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [CANDIDATE]: Term 1 election: Requested vote from peers 3802a0def17a4f7aa3d243566b4c6b85 (127.31.61.129:35605), f8592ad0f4d1422492289daefcc5c76e (127.31.61.130:37511)
I20260812 16:41:22.575013 12773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1d8dfd98a647739e278d448fa851af" candidate_uuid: "2cc5f121322e4c5ca7ded68729141157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3802a0def17a4f7aa3d243566b4c6b85"
I20260812 16:41:22.575034 12919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1d8dfd98a647739e278d448fa851af" candidate_uuid: "2cc5f121322e4c5ca7ded68729141157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8592ad0f4d1422492289daefcc5c76e"
I20260812 16:41:22.575233 12773 raft_consensus.cc:3037] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:22.575227 12919 raft_consensus.cc:3037] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:22.577508 12919 raft_consensus.cc:2454] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc5f121322e4c5ca7ded68729141157 in term 1.
I20260812 16:41:22.577654 12773 raft_consensus.cc:2454] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc5f121322e4c5ca7ded68729141157 in term 1.
I20260812 16:41:22.577941 12988 leader_election.cc:302] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc5f121322e4c5ca7ded68729141157, f8592ad0f4d1422492289daefcc5c76e; no voters:
I20260812 16:41:22.578414 13128 raft_consensus.cc:2781] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:41:22.578869 13128 raft_consensus.cc:686] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 1 LEADER]: Becoming Leader. State: Replica: 2cc5f121322e4c5ca7ded68729141157, State: Running, Role: LEADER
I20260812 16:41:22.579334 13128 consensus_queue.cc:237] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } }
I20260812 16:41:22.584724 12618 catalog_manager.cc:5495] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cc5f121322e4c5ca7ded68729141157 (127.31.61.131). New cstate: current_term: 1 leader_uuid: "2cc5f121322e4c5ca7ded68729141157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc5f121322e4c5ca7ded68729141157" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 } health_report { overall_health: UNKNOWN } } }
W20260812 16:41:22.631520 13101 tablet.cc:2369] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:41:22.642324 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.636592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.31.61.131:34763)
0812 16:41:22.636800 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.636804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.636823 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:41:22.636975 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.636978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.637062 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.637194 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:41:22.637777 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.638523 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.640023 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.640438 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.640593 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.641106 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:41:22.641649 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:41:22.641769 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.641950 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.642027 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 16:41:22.651072 12773 raft_consensus.cc:1261] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [term 1 FOLLOWER]: Refusing update from remote peer 2cc5f121322e4c5ca7ded68729141157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:22.651052 12919 raft_consensus.cc:1261] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [term 1 FOLLOWER]: Refusing update from remote peer 2cc5f121322e4c5ca7ded68729141157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:22.652383 13139 consensus_queue.cc:1035] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8592ad0f4d1422492289daefcc5c76e" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:41:22.652854 13128 consensus_queue.cc:1035] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3802a0def17a4f7aa3d243566b4c6b85" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:41:22.672765 13142 mvcc.cc:205] Tried to move back new op lower bound from 7317720607326887936 to 7317720607049654272. Current Snapshot: MvccSnapshot[applied={T|T < 7317720607326887936}]
I20260812 16:41:22.673194 13143 mvcc.cc:205] Tried to move back new op lower bound from 7317720607326887936 to 7317720607049654272. Current Snapshot: MvccSnapshot[applied={T|T < 7317720607326887936}]
I20260812 16:41:22.688555 12618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:41:22.688890 12618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:41:22.692788 12618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b22869ee243441959cdc182878a6e8ed: Sending DeleteTablet for 3 replicas of tablet 5c1d8dfd98a647739e278d448fa851af
I20260812 16:41:22.694177 13035 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1d8dfd98a647739e278d448fa851af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:22 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:44592
I20260812 16:41:22.694173 12752 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1d8dfd98a647739e278d448fa851af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:22 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:56030
I20260812 16:41:22.694245 12897 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1d8dfd98a647739e278d448fa851af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:22 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:60402
I20260812 16:41:22.699713 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 12657
I20260812 16:41:22.706939 13152 tablet_replica.cc:330] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: stopping tablet replica
I20260812 16:41:22.707676 13152 raft_consensus.cc:2229] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:41:22.708700 13152 raft_consensus.cc:2258] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:22.711375 13152 ts_tablet_manager.cc:1905] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:22.720034 13152 ts_tablet_manager.cc:1918] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:22.720376 13152 log.cc:1196] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/5c1d8dfd98a647739e278d448fa851af
I20260812 16:41:22.721133 13152 ts_tablet_manager.cc:1939] T 5c1d8dfd98a647739e278d448fa851af P 2cc5f121322e4c5ca7ded68729141157: Deleting consensus metadata
I20260812 16:41:22.722056 13154 tablet_replica.cc:330] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: stopping tablet replica
I20260812 16:41:22.722538 13154 raft_consensus.cc:2229] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:41:22.723104 13154 raft_consensus.cc:2258] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:22.723590 12604 catalog_manager.cc:4841] TS 2cc5f121322e4c5ca7ded68729141157 (127.31.61.131:34763): tablet 5c1d8dfd98a647739e278d448fa851af (table test-table [id=0d6edf2c438d4183a38d0f3172489e4e]) successfully deleted
I20260812 16:41:22.728042 13154 ts_tablet_manager.cc:1905] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:22.732584 13153 tablet_replica.cc:330] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: stopping tablet replica
I20260812 16:41:22.733021 13153 raft_consensus.cc:2229] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:41:22.733497 13153 raft_consensus.cc:2258] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:22.735399 13154 ts_tablet_manager.cc:1918] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:22.735579 13154 log.cc:1196] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/5c1d8dfd98a647739e278d448fa851af
I20260812 16:41:22.736155 13154 ts_tablet_manager.cc:1939] T 5c1d8dfd98a647739e278d448fa851af P f8592ad0f4d1422492289daefcc5c76e: Deleting consensus metadata
I20260812 16:41:22.737566 13153 ts_tablet_manager.cc:1905] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:22.738159 12603 catalog_manager.cc:4841] TS f8592ad0f4d1422492289daefcc5c76e (127.31.61.130:37511): tablet 5c1d8dfd98a647739e278d448fa851af (table test-table [id=0d6edf2c438d4183a38d0f3172489e4e]) successfully deleted
I20260812 16:41:22.741621 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.701069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.31.61.129:35605)
0812 16:41:22.701276 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.701282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.719237 (+ 17955us) server_negotiation.cc:407] Connection header received
0812 16:41:22.719460 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.719466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.719559 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.719689 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:22.721547 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.722698 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.725641 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.726608 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.726758 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.733977 (+ 7219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:22.733994 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:22.734013 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:22.734051 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:22.737738 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.738809 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.738824 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.738829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.739055 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.739548 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.739553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.739557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.740143 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:22.740491 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.740828 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.741118 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 16:41:22.742314 12813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:22.836146 13153 ts_tablet_manager.cc:1918] T 5c1d8dfd98a647739e278d448fa851af P 3802a0def17a4f7aa3d243566b4c6b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:22.836469 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 12657
W20260812 16:41:22.859508 12602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.61.129:35605 (error 108)
I20260812 16:41:22.859691 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 12823
I20260812 16:41:22.861910 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.861071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.0.0.1:56812)
0812 16:41:22.861606 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.861726 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":308,"threads_started":1}
W20260812 16:41:22.862200 12602 catalog_manager.cc:4584] TS 3802a0def17a4f7aa3d243566b4c6b85 (127.31.61.129:35605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c1d8dfd98a647739e278d448fa851af: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35605: connect: Connection refused (error 111)
I20260812 16:41:22.876304 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.860548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.31.61.130:37511)
0812 16:41:22.860739 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.860746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.860823 (+ 77us) server_negotiation.cc:407] Connection header received
0812 16:41:22.861048 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.861054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.861159 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.861269 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:22.863162 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.864046 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.865932 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.866345 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:22.866450 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:22.869765 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:22.869782 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:22.869799 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:22.869837 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:22.872826 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.873812 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.873820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.873823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.874198 (+ 375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:22.874644 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:22.874648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:22.874651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:22.875072 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:22.875298 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:22.875806 (+ 508us) server_negotiation.cc:299] Negotiation successful
0812 16:41:22.876023 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:41:22.877038 12960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:22.915946 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.915355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.0.0.1:56814)
0812 16:41:22.915652 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.915773 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 16:41:22.917145 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.916709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.0.0.1:56816)
0812 16:41:22.916876 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.916969 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 16:41:22.968418 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 12823
I20260812 16:41:22.977813 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.977347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.0.0.1:56820)
0812 16:41:22.977557 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.977662 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 16:41:22.978879 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.978530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.0.0.1:56836)
0812 16:41:22.978676 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:22.978746 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 16:41:22.994310 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 12969
I20260812 16:41:23.010506 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:22.995064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.31.61.131:34763)
0812 16:41:22.995229 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:22.995233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:22.995257 (+ 24us) server_negotiation.cc:407] Connection header received
0812 16:41:22.995472 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:22.995477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:22.995554 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:22.995687 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:22.997511 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:22.998298 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:23.000393 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:23.000836 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:23.000934 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:23.004281 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:23.004293 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:23.004309 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:23.004350 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:23.007502 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:23.008388 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:23.008397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:23.008399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:23.008553 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:23.008935 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:23.008938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:23.008940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:23.009307 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:23.009634 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:23.009923 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 16:41:23.010145 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 16:41:23.011485 13095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:23.059829 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:23.059207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.0.0.1:56852)
0812 16:41:23.059472 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:23.059634 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 16:41:23.061244 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:23.060677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.0.0.1:56860)
0812 16:41:23.060923 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:23.061037 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260812 16:41:23.134833 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 12969
I20260812 16:41:23.152825 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 12584
I20260812 16:41:23.154165 12649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:23.268824 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:23.268196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35605 (local address 127.0.0.1:56868)
0812 16:41:23.268426 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:23.268618 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 16:41:23.269091 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 12584
I20260812 16:41:23.289340 31990 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 16:41:23.625408 31990 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 16:41:23.625641 31990 mini_ranger.cc:66] Stopping Ranger...
I20260812 16:41:23.972759 31990 mini_ranger.cc:68] Stopped Ranger
2026-08-12 16:41:23.973 UTC [11628] LOG: received smart shutdown request
2026-08-12 16:41:23.976 UTC [11628] LOG: background worker "logical replication launcher" (PID 11637) exited with exit code 1
2026-08-12 16:41:23.976 UTC [11632] LOG: shutting down
2026-08-12 16:41:24.181 UTC [11628] LOG: database system is shut down
2026-08-12T16:41:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53287 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:41:24 dist-test-slave-f7th krb5kdc[13170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:41:24 dist-test-slave-f7th krb5kdc[13170](info): set up 2 sockets
Aug 12 16:41:24 dist-test-slave-f7th krb5kdc[13170](info): commencing operation
krb5kdc: starting...
W20260812 16:41:25.344637 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:41:25 dist-test-slave-f7th krb5kdc[13170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:41:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:41:25.605324 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:35125
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:35125,127.31.61.189:36025,127.31.61.188:33255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:25.944456 13183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:25.944777 13183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:25.944905 13183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:25.953420 13183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:25.953557 13183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:25.953629 13183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:25.953711 13183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:25.953765 13183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:25.964249 13183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:35125,127.31.61.189:36025,127.31.61.188:33255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:25.965643 13183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:25.967044 13183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:25.979463 13188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:25.979424 13191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:25.979394 13189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:25.980523 13183 server_base.cc:1034] running on GCE node
I20260812 16:41:25.981389 13183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:25.982898 13183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:25.984118 13183 hybrid_clock.cc:648] HybridClock initialized: now 1786552885984053 us; error 46 us; skew 500 ppm
I20260812 16:41:25.984486 13183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:25 dist-test-slave-f7th krb5kdc[13170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552885, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:25.987699 13183 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:41:25.989389 13183 webserver.cc:460] Webserver started at http://127.31.61.190:46287/ using document root <none> and password file <none>
I20260812 16:41:25.990092 13183 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:25.990195 13183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:25.990454 13183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:25.993557 13183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "dde9b59180844e0da2cda5d209d11853"
format_stamp: "Formatted at 2026-08-12 16:41:25 on dist-test-slave-f7th"
server_key: "bf4ede3a82b142198264316fe7db63d8"
server_key_iv: "a20591fa76c00df6291cff8376a972d3"
server_key_version: "encryptionkey@0"
I20260812 16:41:25.994354 13183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "dde9b59180844e0da2cda5d209d11853"
format_stamp: "Formatted at 2026-08-12 16:41:25 on dist-test-slave-f7th"
server_key: "bf4ede3a82b142198264316fe7db63d8"
server_key_iv: "a20591fa76c00df6291cff8376a972d3"
server_key_version: "encryptionkey@0"
I20260812 16:41:25.999790 13183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:41:26.003564 13198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:26.005445 13183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:41:26.005671 13183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "dde9b59180844e0da2cda5d209d11853"
format_stamp: "Formatted at 2026-08-12 16:41:25 on dist-test-slave-f7th"
server_key: "bf4ede3a82b142198264316fe7db63d8"
server_key_iv: "a20591fa76c00df6291cff8376a972d3"
server_key_version: "encryptionkey@0"
I20260812 16:41:26.005875 13183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:26.029601 13183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:26.034447 13183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:26.034760 13183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:26.053819 13183 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:35125
I20260812 16:41:26.053866 13252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:35125 every 8 connection(s)
I20260812 16:41:26.055259 13183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:41:26.059216 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:26.060253 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 13183
I20260812 16:41:26.060508 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:41:26.061357 31990 external_mini_cluster.cc:1428] Setting key 9564f410a89b6833a84e1b45cdf149f2
I20260812 16:41:26.069036 13253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:41:26 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552885, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.61.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.189@KRBTEST.COM" created.
I20260812 16:41:26.094038 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.073157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.31.61.190:35125)
0812 16:41:26.074035 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:26.074050 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:26.074489 (+ 439us) server_negotiation.cc:407] Connection header received
0812 16:41:26.076785 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:26.076792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:26.077050 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:26.077257 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:26.078727 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.080015 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.081447 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.082107 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.082287 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:26.085633 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:26.085642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:26.085655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:26.085690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:26.088633 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:26.089467 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:26.089475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:26.089477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:26.089656 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:26.090339 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:26.090343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:26.090345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:26.091080 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:26.092053 (+ 973us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:26.092568 (+ 515us) server_negotiation.cc:299] Negotiation successful
0812 16:41:26.092881 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":400,"threads_started":1}
I20260812 16:41:26.094094 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.073157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:35125 (local address 127.0.0.1:42558)
0812 16:41:26.074327 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.074367 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:26.076230 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:26.077295 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:26.077302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:26.078167 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:26.078512 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:26.078532 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.080168 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.080172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:26.081294 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:26.081312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.082348 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.082356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:26.082522 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:26.083131 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:26.083161 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:26.085409 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:26.088757 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:26.088769 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:26.088783 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:26.089309 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:26.089784 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:26.089789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:26.089793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:26.090017 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:26.091244 (+ 1227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:26.091268 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:26.092122 (+ 854us) client_negotiation.cc:769] Sending connection context
0812 16:41:26.092440 (+ 318us) client_negotiation.cc:240] Negotiation successful
0812 16:41:26.093061 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":79,"spinlock_wait_cycles":53504,"thread_start_us":727,"threads_started":1}
I20260812 16:41:26.097765 13253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:41:26.099327 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.098887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34548)
0812 16:41:26.099078 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.099193 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 16:41:26.099586 13201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
I20260812 16:41:26.101441 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.101025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34556)
0812 16:41:26.101208 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.101286 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 16:41:26.101655 13253 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:36025: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab.
Entry for principal kudu/127.31.61.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab.
I20260812 16:41:26.122828 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.189:36025
--webserver_interface=127.31.61.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab
--principal=kudu/127.31.61.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:35125,127.31.61.189:36025,127.31.61.188:33255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:41:26.151093 13253 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } attempt: 1
I20260812 16:41:26.152293 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.151827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34572)
0812 16:41:26.152026 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.152155 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 16:41:26.153294 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.152937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34574)
0812 16:41:26.153086 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.153160 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 16:41:26.153478 13253 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:36025: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
I20260812 16:41:26.221743 13253 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } attempt: 2
I20260812 16:41:26.222895 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.222408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34584)
0812 16:41:26.222655 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.222757 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 16:41:26.223899 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.223497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34590)
0812 16:41:26.223683 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.223758 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 16:41:26.224078 13253 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:36025: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
I20260812 16:41:26.315322 13253 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } attempt: 3
I20260812 16:41:26.316706 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.316049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34606)
0812 16:41:26.316429 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.316554 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260812 16:41:26.317775 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.317362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34614)
0812 16:41:26.317547 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.317635 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 16:41:26.318006 13253 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:36025: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
I20260812 16:41:26.461270 13253 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } attempt: 4
I20260812 16:41:26.462723 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.461989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34626)
0812 16:41:26.462365 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.462541 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260812 16:41:26.463807 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.463368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34628)
0812 16:41:26.463524 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.463647 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 16:41:26.464035 13253 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:36025: Network error: Client connection negotiation failed: client connection to 127.31.61.189:36025: connect: Connection refused (error 111)
W20260812 16:41:26.470031 13260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:26.470324 13260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:26.470461 13260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:26.477851 13260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:26.477967 13260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:26.478044 13260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:26.478121 13260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:26.478184 13260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:26.488274 13260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:35125,127.31.61.189:36025,127.31.61.188:33255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab
--principal=kudu/127.31.61.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.189:36025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.61.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:26.489611 13260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:26.490906 13260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:26.502529 13268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:26.502589 13265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:26.502512 13266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:26.503280 13260 server_base.cc:1034] running on GCE node
I20260812 16:41:26.504127 13260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:26.505566 13260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:26.506788 13260 hybrid_clock.cc:648] HybridClock initialized: now 1786552886506736 us; error 46 us; skew 500 ppm
I20260812 16:41:26.507156 13260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:26 dist-test-slave-f7th krb5kdc[13170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552886, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:26.516178 13260 init.cc:377] Logged in from keytab as kudu/127.31.61.189@KRBTEST.COM (short username kudu)
I20260812 16:41:26.517802 13260 webserver.cc:460] Webserver started at http://127.31.61.189:45893/ using document root <none> and password file <none>
I20260812 16:41:26.518507 13260 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:26.518613 13260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:26.518919 13260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:26.522094 13260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/data/instance:
uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7"
format_stamp: "Formatted at 2026-08-12 16:41:26 on dist-test-slave-f7th"
server_key: "cf51cc143dcc5a2332f884fbaf9b8854"
server_key_iv: "ab75eb7c659229a8acd92140e46335ff"
server_key_version: "encryptionkey@0"
I20260812 16:41:26.522908 13260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/wal/instance:
uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7"
format_stamp: "Formatted at 2026-08-12 16:41:26 on dist-test-slave-f7th"
server_key: "cf51cc143dcc5a2332f884fbaf9b8854"
server_key_iv: "ab75eb7c659229a8acd92140e46335ff"
server_key_version: "encryptionkey@0"
I20260812 16:41:26.528393 13260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:41:26.532133 13275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:26.533661 13260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:41:26.533843 13260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/wal
uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7"
format_stamp: "Formatted at 2026-08-12 16:41:26 on dist-test-slave-f7th"
server_key: "cf51cc143dcc5a2332f884fbaf9b8854"
server_key_iv: "ab75eb7c659229a8acd92140e46335ff"
server_key_version: "encryptionkey@0"
I20260812 16:41:26.534049 13260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:26.556337 13260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:26.560822 13260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:26.561138 13260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:26.579807 13260 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.189:36025
I20260812 16:41:26.579819 13326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.189:36025 every 8 connection(s)
I20260812 16:41:26.581398 13260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
I20260812 16:41:26.585610 13327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:26.588615 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 13260
I20260812 16:41:26.588812 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-1/wal/instance
I20260812 16:41:26.589391 31990 external_mini_cluster.cc:1428] Setting key e57be63e17e6700918d2aed185b1a27e
I20260812 16:41:26.594204 13327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:41:26 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552886, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.61.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.188@KRBTEST.COM" created.
I20260812 16:41:26.619180 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.597594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.31.61.190:35125)
0812 16:41:26.598124 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:26.598130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:26.599255 (+ 1125us) server_negotiation.cc:407] Connection header received
0812 16:41:26.601828 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:26.601833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:26.601907 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:26.602014 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:26.604175 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.605002 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.607223 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.607794 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.607937 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:26.612722 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:26.612729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:26.612732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:26.612775 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:26.615505 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:26.616480 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:26.616485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:26.616488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:26.616624 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:26.617082 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:26.617085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:26.617087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:26.617388 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:26.617508 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:26.618527 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0812 16:41:26.618769 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
I20260812 16:41:26.620249 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.598205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:35125 (local address 127.0.0.1:42574)
0812 16:41:26.599065 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.599115 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:26.601470 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:26.602306 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:26.602319 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:26.603040 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:26.603917 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:26.603946 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.605188 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.605194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:26.607033 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:26.607051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.608201 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.608213 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:26.608604 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:26.609477 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:26.609512 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:26.612376 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:26.615779 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:26.615788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:26.615803 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:26.616334 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:26.616747 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:26.616750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:26.616753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:26.616979 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:26.617575 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:26.617586 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:26.618116 (+ 530us) client_negotiation.cc:769] Sending connection context
0812 16:41:26.618382 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 16:41:26.618932 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":358,"threads_started":1}
I20260812 16:41:26.622329 13327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:41:26 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552886, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:41:26.638247 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.623087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34640)
0812 16:41:26.623310 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.623325 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:26.623412 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:26.624435 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:26.624440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:26.624902 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:26.625056 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:26.625065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.626384 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.626389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:26.627219 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:26.627230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.628074 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.628079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:26.628269 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:26.628919 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:26.628941 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:26.631425 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:26.635515 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:26.635519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:26.635522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:26.635964 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:26.636444 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:26.636450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:26.636453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:26.636629 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:26.637435 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:26.637438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:26.637571 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 16:41:26.637683 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 16:41:26.637898 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260812 16:41:26.638624 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.623356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.31.61.189:36025)
0812 16:41:26.624030 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:26.624036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:26.624061 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:41:26.624105 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:26.624111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:26.624291 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:26.624418 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:26.625208 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.626255 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.627346 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.627936 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.628046 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:26.631722 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:26.631739 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:26.631758 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:26.631802 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:26.635319 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:26.636108 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:26.636118 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:26.636121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:26.636270 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:26.636749 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:26.636754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:26.636757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:26.637295 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:26.637530 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:26.637877 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 16:41:26.638177 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":385,"threads_started":1}
I20260812 16:41:26.641168 13327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:41:26.642410 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.641917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49286)
0812 16:41:26.642103 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.642245 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 16:41:26.642764 13276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
I20260812 16:41:26.644419 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.644031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49298)
0812 16:41:26.644222 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.644305 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 16:41:26.644656 13327 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:33255: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab.
Entry for principal kudu/127.31.61.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab.
I20260812 16:41:26.651996 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.188:33255
--webserver_interface=127.31.61.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab
--principal=kudu/127.31.61.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:35125,127.31.61.189:36025,127.31.61.188:33255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:41:26.693989 13327 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } attempt: 1
I20260812 16:41:26.695319 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.694829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49302)
0812 16:41:26.695047 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.695162 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 16:41:26.696530 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.696076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49310)
0812 16:41:26.696262 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.696364 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 16:41:26.696831 13327 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:33255: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
I20260812 16:41:26.763370 13253 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } attempt: 5
I20260812 16:41:26.765069 13327 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } attempt: 2
I20260812 16:41:26.766002 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.765555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49316)
0812 16:41:26.765717 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.765806 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 16:41:26.767071 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.766673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49330)
0812 16:41:26.766832 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.766918 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 16:41:26.767807 13327 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:33255: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Aug 12 16:41:26 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552885, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:41:26.778564 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.764296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.31.61.189:36025)
0812 16:41:26.764496 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:26.764501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:26.764523 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:41:26.764734 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:26.764739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:26.764826 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:26.764931 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:26.765924 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.767042 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.768781 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.769272 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.769377 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:26.773043 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:26.773049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:26.773052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:26.773087 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:26.775798 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:26.776594 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:26.776599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:26.776601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:26.776730 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:26.777142 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:26.777145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:26.777148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:26.777457 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:26.777576 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:26.778097 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 16:41:26.778302 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 16:41:26.778676 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.764165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34656)
0812 16:41:26.764403 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.764426 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:26.764573 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:26.764992 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:26.764997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:26.765504 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:26.765736 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:26.765748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.767194 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.767199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:26.768617 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:26.768634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:26.769431 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:26.769437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:26.769630 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:26.770243 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:26.770266 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:26.772833 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:26.775939 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:26.775946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:26.775950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:26.776446 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:26.776844 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:26.776848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:26.776852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:26.777029 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:26.777586 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:26.777591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:26.777788 (+ 197us) client_negotiation.cc:769] Sending connection context
0812 16:41:26.777954 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 16:41:26.778300 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 16:41:26.779866 13253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:41:26.781010 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.780613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49338)
0812 16:41:26.780793 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.780873 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 16:41:26.782176 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.781783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49344)
0812 16:41:26.781968 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.782063 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 16:41:26.782393 13253 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:33255: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
I20260812 16:41:26.833694 13253 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } attempt: 1
I20260812 16:41:26.834892 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.834408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49350)
0812 16:41:26.834616 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.834725 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 16:41:26.836117 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.835751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49356)
0812 16:41:26.835921 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.836009 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 16:41:26.836326 13253 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:33255: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
I20260812 16:41:26.859057 13327 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } attempt: 3
I20260812 16:41:26.860280 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.859786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49362)
0812 16:41:26.860004 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.860115 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 16:41:26.861312 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.860890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49378)
0812 16:41:26.861120 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.861196 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260812 16:41:26.861531 13327 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:33255: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
I20260812 16:41:26.904569 13253 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } attempt: 2
I20260812 16:41:26.905794 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.905301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49380)
0812 16:41:26.905535 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.905630 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 16:41:26.906773 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:26.906436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49394)
0812 16:41:26.906584 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:26.906659 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 16:41:26.906978 13253 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:33255: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
W20260812 16:41:26.986086 13335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:26.986394 13335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:26.986532 13335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:26.993959 13335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:26.994079 13335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:26.994155 13335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:26.994228 13335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:26.994284 13335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:27.004379 13335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:35125,127.31.61.189:36025,127.31.61.188:33255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab
--principal=kudu/127.31.61.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.188:33255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.61.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:27.004750 13327 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } attempt: 4
I20260812 16:41:27.005766 13335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:27.005846 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.005346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49406)
0812 16:41:27.005601 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:27.005688 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260812 16:41:27.006906 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.006525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49420)
0812 16:41:27.006704 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:27.006790 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 16:41:27.007133 13327 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:33255: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
I20260812 16:41:27.007342 13335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 16:41:27.013161 13253 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } attempt: 3
I20260812 16:41:27.014269 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.013782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49430)
0812 16:41:27.013988 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:27.014091 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 16:41:27.015455 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.015027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49432)
0812 16:41:27.015239 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:27.015324 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260812 16:41:27.015705 13253 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:33255: Network error: Client connection negotiation failed: client connection to 127.31.61.188:33255: connect: Connection refused (error 111)
W20260812 16:41:27.019302 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:27.019374 13342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:27.019726 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:27.020220 13335 server_base.cc:1034] running on GCE node
I20260812 16:41:27.021021 13335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:27.022522 13335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:27.023708 13335 hybrid_clock.cc:648] HybridClock initialized: now 1786552887023662 us; error 38 us; skew 500 ppm
I20260812 16:41:27.024067 13335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:27 dist-test-slave-f7th krb5kdc[13170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552887, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:27.027175 13335 init.cc:377] Logged in from keytab as kudu/127.31.61.188@KRBTEST.COM (short username kudu)
I20260812 16:41:27.028764 13335 webserver.cc:460] Webserver started at http://127.31.61.188:35301/ using document root <none> and password file <none>
I20260812 16:41:27.029461 13335 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:27.029585 13335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:27.029868 13335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:27.032994 13335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/data/instance:
uuid: "90bcd974d0004e1dab27243570f26d5c"
format_stamp: "Formatted at 2026-08-12 16:41:27 on dist-test-slave-f7th"
server_key: "83a18b389be23d52f34fe90fd5729864"
server_key_iv: "cace4c54aee045d52aa299f1e80baf69"
server_key_version: "encryptionkey@0"
I20260812 16:41:27.033967 13335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/wal/instance:
uuid: "90bcd974d0004e1dab27243570f26d5c"
format_stamp: "Formatted at 2026-08-12 16:41:27 on dist-test-slave-f7th"
server_key: "83a18b389be23d52f34fe90fd5729864"
server_key_iv: "cace4c54aee045d52aa299f1e80baf69"
server_key_version: "encryptionkey@0"
I20260812 16:41:27.039671 13335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:41:27.043761 13351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:27.045387 13335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 16:41:27.045574 13335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/wal
uuid: "90bcd974d0004e1dab27243570f26d5c"
format_stamp: "Formatted at 2026-08-12 16:41:27 on dist-test-slave-f7th"
server_key: "83a18b389be23d52f34fe90fd5729864"
server_key_iv: "cace4c54aee045d52aa299f1e80baf69"
server_key_version: "encryptionkey@0"
I20260812 16:41:27.045776 13335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:27.062247 13335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:27.066752 13335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:27.067023 13335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:27.083760 13335 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.188:33255
I20260812 16:41:27.083778 13402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.188:33255 every 8 connection(s)
I20260812 16:41:27.085186 13335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
I20260812 16:41:27.088908 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:27.095533 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 13335
I20260812 16:41:27.095777 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786552676633950-31990-0/minicluster-data/master-2/wal/instance
I20260812 16:41:27.095333 13403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:41:27.096221 31990 external_mini_cluster.cc:1428] Setting key a98ba112b1c81778d965c325ff58b24e
Aug 12 16:41:27 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552887, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
Aug 12 16:41:27 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
I20260812 16:41:27.115742 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.098392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.31.61.190:35125)
0812 16:41:27.098594 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.098601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.099323 (+ 722us) server_negotiation.cc:407] Connection header received
0812 16:41:27.101275 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.101282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.101389 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.101503 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.102853 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.103561 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.105253 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.105875 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.105967 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.109935 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.109940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.109943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.109977 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.112736 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.113467 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.113471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.113474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.113608 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.114019 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.114023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.114025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.114294 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.114441 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.115139 (+ 698us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.115378 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:41:27.116890 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.098007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.31.61.188:33255)
0812 16:41:27.099218 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.099226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.099270 (+ 44us) server_negotiation.cc:407] Connection header received
0812 16:41:27.100949 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.100973 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.101240 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.101398 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.102871 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.104270 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.105862 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.106745 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.106873 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.110977 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.110986 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.110999 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.111031 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.113741 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.114437 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.114443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.114445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.114573 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.114928 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.114933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.114936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.115265 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.115549 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.115895 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.116126 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":999,"thread_start_us":716,"threads_started":1}
I20260812 16:41:27.116923 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.098280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:35125 (local address 127.0.0.1:42584)
0812 16:41:27.099181 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:27.099213 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:27.100978 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:27.101602 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:27.101609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:27.102271 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:27.102665 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.102687 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.103752 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.103758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.105116 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.105128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.106127 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.106134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.106507 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.107312 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:27.107336 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:27.109729 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:27.112865 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.112878 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.112892 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.113351 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.113716 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.113721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.113726 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.113913 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.114419 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:27.114428 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:27.114779 (+ 351us) client_negotiation.cc:769] Sending connection context
0812 16:41:27.115026 (+ 247us) client_negotiation.cc:240] Negotiation successful
0812 16:41:27.115540 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":66,"thread_start_us":444,"threads_started":1}
I20260812 16:41:27.118602 13403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:41:27 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552887, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
Aug 12 16:41:27 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:27.132085 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.119234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:36025 (local address 127.0.0.1:34658)
0812 16:41:27.119365 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:27.119381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:27.119458 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:27.120164 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:27.120168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:27.120539 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:27.120682 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.120690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.121648 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.121654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.122516 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.122527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.123195 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.123200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.123420 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.123948 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:27.123965 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:27.126132 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:27.129504 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.129509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.129513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.130032 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.130474 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.130479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.130482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.130666 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.131227 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:27.131232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:27.131404 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 16:41:27.131552 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 16:41:27.131833 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 16:41:27.132336 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.119687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.31.61.189:36025)
0812 16:41:27.119892 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.119898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.119917 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:41:27.119976 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.119980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.120040 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.120123 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.120823 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.121519 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.122646 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.123064 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.123170 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.126323 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.126328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.126331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.126364 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.129362 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.130170 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.130177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.130180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.130355 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.130804 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.130809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.130811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.131103 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.131229 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.131716 (+ 487us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.131965 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 16:41:27.133258 13403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:41:27.134382 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.119863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.31.61.190:35125)
0812 16:41:27.120019 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.120024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.120080 (+ 56us) server_negotiation.cc:407] Connection header received
0812 16:41:27.120241 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.120246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.120313 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.120391 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.121564 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.122228 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.123469 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.123938 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.124050 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.128568 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.128573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.128576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.128611 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.131458 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.132319 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.132325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.132327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.132478 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.132919 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.132923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.132926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.133268 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.133514 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.133840 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.134072 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 16:41:27 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552887, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
Aug 12 16:41:27 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:41:27.148212 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.134360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.31.61.188:33255)
0812 16:41:27.134537 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.134543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.134559 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:41:27.134602 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.134606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.134671 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.134768 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.135664 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.136559 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.138190 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.138880 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.139000 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.142491 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.142498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.142503 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.142539 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.145545 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.146283 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.146287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.146289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.146423 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.146840 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.146845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.146847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.147127 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.147271 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.147644 (+ 373us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.147852 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 16:41:27.148571 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.134126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49450)
0812 16:41:27.134308 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:27.134326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:27.134430 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:27.134809 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:27.134813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:27.135284 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:27.135501 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.135513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.136695 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.136701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.138058 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.138072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.139026 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.139031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.139176 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.139719 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:27.139742 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:27.142284 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:27.145664 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.145669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.145673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.146155 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.146570 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.146575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.146578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.146740 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.147239 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:27.147244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:27.147386 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 16:41:27.147524 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 16:41:27.148005 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":135}
I20260812 16:41:27.152563 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.136130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.31.61.189:36025)
0812 16:41:27.136295 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.136301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.136318 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:41:27.136385 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.136390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.136453 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.136539 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.138183 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.139254 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.140967 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.141560 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.141684 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.146721 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.146727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.146730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.146763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.149643 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.150453 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.150458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.150461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.150628 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.151046 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.151051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.151055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.151397 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.151711 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.152009 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.152242 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:41:27.154333 13403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c: Bootstrap starting.
I20260812 16:41:27.157409 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:41:27.157864 13403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:27.158888 13403 log.cc:824] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:27.161628 13403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c: No bootstrap required, opened a new log
I20260812 16:41:27.174054 13403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } }
I20260812 16:41:27.174726 13403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:27.174896 13403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bcd974d0004e1dab27243570f26d5c, State: Initialized, Role: FOLLOWER
I20260812 16:41:27.175828 13403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } }
I20260812 16:41:27.183041 13416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.183859 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.162155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.31.61.190:35125)
0812 16:41:27.162355 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.162362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.163212 (+ 850us) server_negotiation.cc:407] Connection header received
0812 16:41:27.163536 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.163542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.163691 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.163798 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.166651 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.167966 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.170875 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.171908 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.172038 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.175985 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.175992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.175997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.176059 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.179750 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.181054 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.181061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.181065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.181268 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.181760 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.181766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.181769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.182187 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.182465 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.183263 (+ 798us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.183501 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:41:27.184414 13416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:41:27.185757 13403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:41:27.191444 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.163866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.31.61.189:36025)
0812 16:41:27.164051 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.164057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.172192 (+ 8135us) server_negotiation.cc:407] Connection header received
0812 16:41:27.172481 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.172488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.172582 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.172688 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.175207 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.176455 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.179052 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.179793 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.179911 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.184753 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.184761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.184765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.184810 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.188003 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.188887 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.188893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.188897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.189073 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.189560 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.189563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.189565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.189859 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.190092 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.190848 (+ 756us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.191090 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:41:27.193547 13253 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } attempt: 4
I20260812 16:41:27.201521 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.164320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.31.61.188:33255)
0812 16:41:27.166535 (+ 2215us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.166542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.183874 (+ 17332us) server_negotiation.cc:407] Connection header received
0812 16:41:27.184153 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.184159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.184242 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.184310 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.185762 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.186571 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.188537 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.189365 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.189471 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.192763 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.192770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.192774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.192819 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.196545 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.197418 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.197427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.197431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.197635 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.198087 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.198092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.198096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.199997 (+ 1901us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.200282 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.200833 (+ 551us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.201108 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2052}
W20260812 16:41:27.206426 13427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:41:27.206700 13427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 12 16:41:27 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552885, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
I20260812 16:41:27.196617 13335 master_runner.cc:186] Error getting master registration for 127.31.61.190:35125: OK, instance_id { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" instance_seqno: 1786552886048305 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:41:27.210368 13335 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 16:41:27.217435 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.194123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49474)
0812 16:41:27.194523 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:27.194552 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:27.194660 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:27.202076 (+ 7416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:27.202082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:27.202735 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:27.202941 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.202957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.204419 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.204424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.205807 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.205824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.206873 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.206879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.207062 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.207678 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:27.207700 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:27.210386 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:27.213837 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.213843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.213847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.214273 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.214653 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.214658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.214662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.214791 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.215426 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:27.215432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:27.215641 (+ 209us) client_negotiation.cc:769] Sending connection context
0812 16:41:27.215801 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 16:41:27.217046 (+ 1245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275}
I20260812 16:41:27.218343 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.194519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.31.61.188:33255)
0812 16:41:27.201765 (+ 7246us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.201770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.201786 (+ 16us) server_negotiation.cc:407] Connection header received
0812 16:41:27.201828 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.201833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.201927 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.202034 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.203092 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.204251 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.206015 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.206686 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.207379 (+ 693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.210602 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.210609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.210613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.210649 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.213701 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.214391 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.214396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.214399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.214537 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.214895 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.214900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.214902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.215253 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.215410 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.217745 (+ 2335us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.217971 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7150,"thread_start_us":12485,"threads_started":1}
I20260812 16:41:27.223322 13253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853: Bootstrap starting.
I20260812 16:41:27.226397 13253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:27.227416 13253 log.cc:824] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:27.230856 13253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853: No bootstrap required, opened a new log
I20260812 16:41:27.239029 13253 raft_consensus.cc:348] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } }
I20260812 16:41:27.239449 13253 raft_consensus.cc:374] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:27.239583 13253 raft_consensus.cc:729] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde9b59180844e0da2cda5d209d11853, State: Initialized, Role: FOLLOWER
I20260812 16:41:27.240279 13253 consensus_queue.cc:260] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } }
I20260812 16:41:27.241216 13431 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.241525 13431 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:41:27.242630 13253 sys_catalog.cc:564] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 16:41:27.253724 13442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:41:27.253953 13442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:41:27.253010 13183 master_runner.cc:186] Error getting master registration for 127.31.61.189:36025: OK, instance_id { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" instance_seqno: 1786552886574149 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:41:27.262730 13183 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 16:41:27.287883 13204 tablet.cc:2369] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:41:27.306370 13327 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } attempt: 5
Aug 12 16:41:27 dist-test-slave-f7th krb5kdc[13170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552886, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
W20260812 16:41:27.319255 13357 tablet.cc:2369] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:41:27.322563 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.307081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:33255 (local address 127.0.0.1:49484)
0812 16:41:27.307382 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:27.307402 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:27.307533 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:27.307882 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:27.307886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:27.308324 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:27.308494 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.308503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.309669 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.309675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.310735 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:27.310749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.312884 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.312890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:27.313144 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.314285 (+ 1141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:27.314309 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:27.316445 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:27.320035 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.320041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.320044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.320495 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.320869 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:27.320873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:27.320875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:27.321015 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:27.321579 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:27.321584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:27.321721 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 16:41:27.321848 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 16:41:27.322170 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260812 16:41:27.322595 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:27.307165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.31.61.188:33255)
0812 16:41:27.307388 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:27.307394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:27.307462 (+ 68us) server_negotiation.cc:407] Connection header received
0812 16:41:27.307665 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:27.307671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:27.307735 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:27.307826 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:27.308658 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.309502 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.310875 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:27.311563 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:27.312199 (+ 636us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:27.316658 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:27.316664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:27.316667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:27.316717 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:27.319842 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.320618 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.320623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.320627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.320754 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:27.321132 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:27.321138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:27.321142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:27.321452 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:27.321577 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:27.321982 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 16:41:27.322227 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 16:41:27.327288 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7: Bootstrap starting.
I20260812 16:41:27.330623 13327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:27.332060 13327 log.cc:824] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:27.337725 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7: No bootstrap required, opened a new log
I20260812 16:41:27.346421 13327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } }
I20260812 16:41:27.346797 13327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:27.346913 13327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f919a3a5d94a0ab5734b6b0e21d8e7, State: Initialized, Role: FOLLOWER
I20260812 16:41:27.347550 13327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } }
I20260812 16:41:27.348555 13445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.349041 13445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:41:27.349953 13327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:41:27.372385 13260 master_runner.cc:263] No leader master found from master 00f919a3a5d94a0ab5734b6b0e21d8e7
I20260812 16:41:27.372565 13260 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 16:41:27.376206 13456 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:41:27.376466 13456 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:41:27.398185 13416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:27.398553 13416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } }
I20260812 16:41:27.400498 13416 leader_election.cc:288] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dde9b59180844e0da2cda5d209d11853 (127.31.61.190:35125), 00f919a3a5d94a0ab5734b6b0e21d8e7 (127.31.61.189:36025)
I20260812 16:41:27.401674 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90bcd974d0004e1dab27243570f26d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde9b59180844e0da2cda5d209d11853" is_pre_election: true
I20260812 16:41:27.402351 13227 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90bcd974d0004e1dab27243570f26d5c in term 0.
I20260812 16:41:27.403081 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90bcd974d0004e1dab27243570f26d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" is_pre_election: true
I20260812 16:41:27.404256 13354 leader_election.cc:302] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90bcd974d0004e1dab27243570f26d5c, dde9b59180844e0da2cda5d209d11853; no voters:
I20260812 16:41:27.404772 13416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:41:27.404938 13416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:41:27.405045 13416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:27.408560 13416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } }
I20260812 16:41:27.409330 13302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90bcd974d0004e1dab27243570f26d5c in term 0.
I20260812 16:41:27.411898 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90bcd974d0004e1dab27243570f26d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde9b59180844e0da2cda5d209d11853"
I20260812 16:41:27.412163 13227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:27.415946 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90bcd974d0004e1dab27243570f26d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7"
I20260812 16:41:27.416209 13302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:27.422849 13416 leader_election.cc:288] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [CANDIDATE]: Term 1 election: Requested vote from peers dde9b59180844e0da2cda5d209d11853 (127.31.61.190:35125), 00f919a3a5d94a0ab5734b6b0e21d8e7 (127.31.61.189:36025)
I20260812 16:41:27.423960 13302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90bcd974d0004e1dab27243570f26d5c in term 1.
I20260812 16:41:27.424343 13227 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90bcd974d0004e1dab27243570f26d5c in term 1.
I20260812 16:41:27.426899 13354 leader_election.cc:302] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90bcd974d0004e1dab27243570f26d5c, dde9b59180844e0da2cda5d209d11853; no voters:
I20260812 16:41:27.427350 13416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:41:27.436038 13416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [term 1 LEADER]: Becoming Leader. State: Replica: 90bcd974d0004e1dab27243570f26d5c, State: Running, Role: LEADER
I20260812 16:41:27.436834 13416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } }
I20260812 16:41:27.444730 13458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90bcd974d0004e1dab27243570f26d5c. Latest consensus state: current_term: 1 leader_uuid: "90bcd974d0004e1dab27243570f26d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.445410 13458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:27.456169 13460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:41:27.458781 13460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:41:27.480682 13302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [term 1 FOLLOWER]: Refusing update from remote peer 90bcd974d0004e1dab27243570f26d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:27.482069 13458 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:41:27.490075 13227 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [term 1 FOLLOWER]: Refusing update from remote peer 90bcd974d0004e1dab27243570f26d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:27.491230 13458 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:41:27.527786 13431 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90bcd974d0004e1dab27243570f26d5c. Latest consensus state: current_term: 1 leader_uuid: "90bcd974d0004e1dab27243570f26d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.528185 13431 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:41:27.537071 13461 mvcc.cc:205] Tried to move back new op lower bound from 7317720627107672064 to 7317720626949066752. Current Snapshot: MvccSnapshot[applied={T|T < 7317720627107672064}]
I20260812 16:41:27.551419 13463 mvcc.cc:205] Tried to move back new op lower bound from 7317720627107672064 to 7317720626949066752. Current Snapshot: MvccSnapshot[applied={T|T < 7317720627107672064}]
I20260812 16:41:27.563992 13460 catalog_manager.cc:1347] Generated new cluster ID: aab45d499db24c35ba8cc93c1942722c
I20260812 16:41:27.564136 13460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260812 16:41:27.567888 13281 tablet.cc:2369] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:41:27.579495 13416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90bcd974d0004e1dab27243570f26d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.579885 13416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:27.580569 13431 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90bcd974d0004e1dab27243570f26d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.580833 13431 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:41:27.589269 13445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90bcd974d0004e1dab27243570f26d5c. Latest consensus state: current_term: 1 leader_uuid: "90bcd974d0004e1dab27243570f26d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.589594 13445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:41:27.600683 13458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90bcd974d0004e1dab27243570f26d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.602144 13458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:27.614266 13445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90bcd974d0004e1dab27243570f26d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde9b59180844e0da2cda5d209d11853" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35125 } } peers { permanent_uuid: "00f919a3a5d94a0ab5734b6b0e21d8e7" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 36025 } } peers { permanent_uuid: "90bcd974d0004e1dab27243570f26d5c" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 33255 } } }
I20260812 16:41:27.614537 13445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:41:27.639154 13460 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:41:27.641350 13460 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:41:27.659513 13460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90bcd974d0004e1dab27243570f26d5c: Generated new TSK 0
I20260812 16:41:27.660840 13460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:41:28.256058 13442 catalog_manager.cc:1259] Loaded cluster ID: aab45d499db24c35ba8cc93c1942722c
I20260812 16:41:28.256170 13442 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853: loading cluster ID for follower catalog manager: success
I20260812 16:41:28.258792 13442 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853: acquiring CA information for follower catalog manager: success
I20260812 16:41:28.260807 13442 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dde9b59180844e0da2cda5d209d11853: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:41:28.378504 13456 catalog_manager.cc:1259] Loaded cluster ID: aab45d499db24c35ba8cc93c1942722c
I20260812 16:41:28.378611 13456 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7: loading cluster ID for follower catalog manager: success
I20260812 16:41:28.381137 13456 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7: acquiring CA information for follower catalog manager: success
I20260812 16:41:28.382998 13456 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00f919a3a5d94a0ab5734b6b0e21d8e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:41:28.798391 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 13183
I20260812 16:41:28.799839 13248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:28.911969 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 13183
I20260812 16:41:28.930425 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 13260
I20260812 16:41:28.931546 13322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:29.037540 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 13260
I20260812 16:41:29.055893 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 13335
I20260812 16:41:29.057061 13398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:29.165263 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 13335
2026-08-12T16:41:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4863 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:41:29 dist-test-slave-f7th krb5kdc[13482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:41:29 dist-test-slave-f7th krb5kdc[13482](info): set up 2 sockets
Aug 12 16:41:29 dist-test-slave-f7th krb5kdc[13482](info): commencing operation
krb5kdc: starting...
W20260812 16:41:30.205317 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:41:30 dist-test-slave-f7th krb5kdc[13482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:41:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:41:30.452718 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:39149
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:39149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:30.805192 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:30.805501 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:30.805634 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:30.812973 13495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:30.813112 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:30.813198 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:30.813275 13495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:30.813336 13495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:30.823649 13495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:39149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:39149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:30.825050 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:30.826643 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:30.839984 13500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:30.840255 13503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:30.840308 13501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:30.840880 13495 server_base.cc:1034] running on GCE node
I20260812 16:41:30.841724 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:30.843165 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:30.844436 13495 hybrid_clock.cc:648] HybridClock initialized: now 1786552890844378 us; error 40 us; skew 500 ppm
I20260812 16:41:30.844965 13495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:30 dist-test-slave-f7th krb5kdc[13482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:30.849391 13495 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:41:30.851228 13495 webserver.cc:460] Webserver started at http://127.31.61.190:42363/ using document root <none> and password file <none>
I20260812 16:41:30.852041 13495 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:30.852160 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:30.852490 13495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:30.855842 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "2a90d5eecc5b47aba5566b32521c6aa6"
format_stamp: "Formatted at 2026-08-12 16:41:30 on dist-test-slave-f7th"
server_key: "b1207a98e651f6c38735c66c00b9297a"
server_key_iv: "ca849ba9c954232516b6240bc96e0f1a"
server_key_version: "encryptionkey@0"
I20260812 16:41:30.856679 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "2a90d5eecc5b47aba5566b32521c6aa6"
format_stamp: "Formatted at 2026-08-12 16:41:30 on dist-test-slave-f7th"
server_key: "b1207a98e651f6c38735c66c00b9297a"
server_key_iv: "ca849ba9c954232516b6240bc96e0f1a"
server_key_version: "encryptionkey@0"
I20260812 16:41:30.862119 13495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 16:41:30.865694 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:30.867120 13495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:41:30.867296 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "2a90d5eecc5b47aba5566b32521c6aa6"
format_stamp: "Formatted at 2026-08-12 16:41:30 on dist-test-slave-f7th"
server_key: "b1207a98e651f6c38735c66c00b9297a"
server_key_iv: "ca849ba9c954232516b6240bc96e0f1a"
server_key_version: "encryptionkey@0"
I20260812 16:41:30.867482 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:30.904389 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:30.908581 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:30.908854 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:30.926942 13495 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:39149
I20260812 16:41:30.927006 13561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:39149 every 8 connection(s)
I20260812 16:41:30.928354 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:41:30.932102 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:30.937573 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 13495
I20260812 16:41:30.937780 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:41:30.938437 31990 external_mini_cluster.cc:1428] Setting key 9b0a50b2cc7bdce9ad1fec462a930350
I20260812 16:41:30.943156 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6: Bootstrap starting.
I20260812 16:41:30.946760 13562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:30.948038 13562 log.cc:824] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:30.951515 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6: No bootstrap required, opened a new log
Aug 12 16:41:30 dist-test-slave-f7th krb5kdc[13482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:30.961650 13562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a90d5eecc5b47aba5566b32521c6aa6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39149 } }
I20260812 16:41:30.962162 13562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:30.962283 13562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a90d5eecc5b47aba5566b32521c6aa6, State: Initialized, Role: FOLLOWER
I20260812 16:41:30.962972 13562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a90d5eecc5b47aba5566b32521c6aa6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39149 } }
I20260812 16:41:30.963280 13562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:41:30.963430 13562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:41:30.963644 13562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:30.963960 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:30.940359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.31.61.190:39149)
0812 16:41:30.941257 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:30.941271 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:30.941312 (+ 41us) server_negotiation.cc:407] Connection header received
0812 16:41:30.943017 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:30.943041 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:30.943435 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:30.943787 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:30.945613 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:30.947014 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:30.948911 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:30.949761 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:30.949911 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:30.954944 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:30.954968 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:30.954990 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:30.955041 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:30.959069 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:30.959935 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:30.959956 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:30.959966 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:30.960193 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:30.960552 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:30.960558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:30.960561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:30.961306 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:30.961589 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:30.962020 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 16:41:30.962485 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":426,"threads_started":1}
I20260812 16:41:30.966630 13562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a90d5eecc5b47aba5566b32521c6aa6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39149 } }
I20260812 16:41:30.967172 13562 leader_election.cc:302] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a90d5eecc5b47aba5566b32521c6aa6; no voters:
I20260812 16:41:30.967869 13562 leader_election.cc:288] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:41:30.968047 13567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:41:30.968390 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:41:30.970129 13562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:41:30.972368 13567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 2a90d5eecc5b47aba5566b32521c6aa6, State: Running, Role: LEADER
I20260812 16:41:30.973889 13567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a90d5eecc5b47aba5566b32521c6aa6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39149 } }
I20260812 16:41:30.977483 13569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a90d5eecc5b47aba5566b32521c6aa6. Latest consensus state: current_term: 1 leader_uuid: "2a90d5eecc5b47aba5566b32521c6aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a90d5eecc5b47aba5566b32521c6aa6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39149 } } }
I20260812 16:41:30.977516 13568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a90d5eecc5b47aba5566b32521c6aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a90d5eecc5b47aba5566b32521c6aa6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 39149 } } }
I20260812 16:41:30.977943 13569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:30.978024 13568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6 [sys.catalog]: This master's current role is: LEADER
W20260812 16:41:30.982606 13581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:41:30.982720 13581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:41:30.982856 13582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:41:30.984532 13582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:41:30.993005 13582 catalog_manager.cc:1347] Generated new cluster ID: 73a4d2fc4faa4d42bcd3dcf815e8364d
I20260812 16:41:30.993119 13582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:41:31.008764 13582 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:41:31.010125 13582 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:41:31.017151 13582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a90d5eecc5b47aba5566b32521c6aa6: Generated new TSK 0
I20260812 16:41:31.018051 13582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:41:31.031417 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 13495
I20260812 16:41:31.032847 13557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:31.126953 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 13495
2026-08-12T16:41:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1957 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:41:31 dist-test-slave-f7th krb5kdc[13590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:41:31 dist-test-slave-f7th krb5kdc[13590](info): set up 2 sockets
Aug 12 16:41:31 dist-test-slave-f7th krb5kdc[13590](info): commencing operation
krb5kdc: starting...
W20260812 16:41:32.163856 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:41:32 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:41:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:41:32.424784 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36469
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:32.767258 13603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:32.767580 13603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:32.767755 13603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:32.775236 13603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:32.775362 13603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:32.775414 13603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:32.775506 13603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:32.775615 13603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:32.786221 13603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:32.787626 13603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:32.789002 13603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:32.800423 13612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:32.800486 13609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:32.800491 13610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:32.801302 13603 server_base.cc:1034] running on GCE node
I20260812 16:41:32.802155 13603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:32.803723 13603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:32.804971 13603 hybrid_clock.cc:648] HybridClock initialized: now 1786552892804912 us; error 45 us; skew 500 ppm
I20260812 16:41:32.805363 13603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:32 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552892, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:32.813603 13603 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:41:32.815255 13603 webserver.cc:460] Webserver started at http://127.31.61.190:45713/ using document root <none> and password file <none>
I20260812 16:41:32.816030 13603 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:32.816159 13603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:32.816462 13603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:32.819582 13603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "2d8a13c711934aed9d25ccd53efe8324"
format_stamp: "Formatted at 2026-08-12 16:41:32 on dist-test-slave-f7th"
server_key: "e5a7e0d90a865b9d8146d69cda5d222d"
server_key_iv: "dd2bb86751438974ce293ad409a08e2c"
server_key_version: "encryptionkey@0"
I20260812 16:41:32.820458 13603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "2d8a13c711934aed9d25ccd53efe8324"
format_stamp: "Formatted at 2026-08-12 16:41:32 on dist-test-slave-f7th"
server_key: "e5a7e0d90a865b9d8146d69cda5d222d"
server_key_iv: "dd2bb86751438974ce293ad409a08e2c"
server_key_version: "encryptionkey@0"
I20260812 16:41:32.825878 13603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 16:41:32.829704 13619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:32.831117 13603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:41:32.831285 13603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "2d8a13c711934aed9d25ccd53efe8324"
format_stamp: "Formatted at 2026-08-12 16:41:32 on dist-test-slave-f7th"
server_key: "e5a7e0d90a865b9d8146d69cda5d222d"
server_key_iv: "dd2bb86751438974ce293ad409a08e2c"
server_key_version: "encryptionkey@0"
I20260812 16:41:32.831475 13603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:32.860597 13603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:32.865027 13603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:32.865319 13603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:32.883510 13603 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36469
I20260812 16:41:32.883548 13670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36469 every 8 connection(s)
I20260812 16:41:32.884920 13603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:41:32.888731 13671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:32.889813 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 13603
I20260812 16:41:32.890009 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:41:32.890663 31990 external_mini_cluster.cc:1428] Setting key cf8dcaf320ac71b7ab6cfcb6f0770807
I20260812 16:41:32.902690 13671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Bootstrap starting.
Aug 12 16:41:32 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:32.906595 13671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:32.907737 13671 log.cc:824] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:32.910974 13671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: No bootstrap required, opened a new log
I20260812 16:41:32.914995 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:32.892630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.31.61.190:36469)
0812 16:41:32.893444 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:32.893456 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:32.893501 (+ 45us) server_negotiation.cc:407] Connection header received
0812 16:41:32.895062 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:32.895084 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:32.895460 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:32.895808 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:32.897488 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:32.898816 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:32.900682 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:32.901384 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:32.901532 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:32.906435 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:32.906443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:32.906466 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:32.906515 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:32.909985 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:32.910763 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:32.910771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:32.910780 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:32.911014 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:32.911487 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:32.911492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:32.911495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:32.912259 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:32.912607 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:32.913062 (+ 455us) server_negotiation.cc:299] Negotiation successful
0812 16:41:32.913552 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":389,"threads_started":1}
I20260812 16:41:32.920518 13671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:32.920869 13671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:32.920943 13671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8a13c711934aed9d25ccd53efe8324, State: Initialized, Role: FOLLOWER
I20260812 16:41:32.921496 13671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:32.921706 13671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:41:32.921808 13671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:41:32.921932 13671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:32.924180 13671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:32.924654 13671 leader_election.cc:302] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d8a13c711934aed9d25ccd53efe8324; no voters:
I20260812 16:41:32.925259 13671 leader_election.cc:288] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:41:32.925421 13676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:41:32.926103 13676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 1 LEADER]: Becoming Leader. State: Replica: 2d8a13c711934aed9d25ccd53efe8324, State: Running, Role: LEADER
I20260812 16:41:32.926627 13676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:32.927372 13671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:41:32.929452 13677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d8a13c711934aed9d25ccd53efe8324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } } }
I20260812 16:41:32.929975 13677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:32.929593 13678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d8a13c711934aed9d25ccd53efe8324. Latest consensus state: current_term: 1 leader_uuid: "2d8a13c711934aed9d25ccd53efe8324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } } }
I20260812 16:41:32.930452 13678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:32.934388 13685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:41:32.938405 13685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:41:32.946962 13685 catalog_manager.cc:1347] Generated new cluster ID: 7f6c91398ccc40b6b4117472079e3b4b
I20260812 16:41:32.947093 13685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:41:32.966001 13685 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:41:32.967324 13685 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:41:32.973229 13685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Generated new TSK 0
I20260812 16:41:32.974098 13685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:41:33.040328 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36469
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:33.378360 13697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:33.378641 13697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:33.378759 13697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:33.386049 13697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:33.386161 13697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:33.386307 13697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:41:33.396215 13697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:33.397539 13697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:33.398857 13697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:33.411942 13702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:33.411870 13703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:33.411892 13705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:33.412431 13697 server_base.cc:1034] running on GCE node
I20260812 16:41:33.413069 13697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:33.413995 13697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:33.415213 13697 hybrid_clock.cc:648] HybridClock initialized: now 1786552893415147 us; error 54 us; skew 500 ppm
I20260812 16:41:33.415547 13697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:33 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:33.418778 13697 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:41:33.420506 13697 webserver.cc:460] Webserver started at http://127.31.61.129:45775/ using document root <none> and password file <none>
I20260812 16:41:33.421250 13697 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:33.421376 13697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:33.421697 13697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:33.424932 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "4f699173dce3457ba9d34619d8d932e8"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "ec131f77babb350648445818f835a9cd"
server_key_iv: "d1c11db5cf62d38cdf6460b3214c765a"
server_key_version: "encryptionkey@0"
I20260812 16:41:33.425753 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "4f699173dce3457ba9d34619d8d932e8"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "ec131f77babb350648445818f835a9cd"
server_key_iv: "d1c11db5cf62d38cdf6460b3214c765a"
server_key_version: "encryptionkey@0"
I20260812 16:41:33.431301 13697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:41:33.435252 13712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:33.436805 13697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:41:33.436978 13697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "4f699173dce3457ba9d34619d8d932e8"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "ec131f77babb350648445818f835a9cd"
server_key_iv: "d1c11db5cf62d38cdf6460b3214c765a"
server_key_version: "encryptionkey@0"
I20260812 16:41:33.437176 13697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:33.463145 13697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:33.467692 13697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:33.467983 13697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:33.469022 13697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:33.470484 13697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:33.470559 13697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:33.470660 13697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:33.470716 13697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:33.501771 13697 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:35119
I20260812 16:41:33.501821 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:35119 every 8 connection(s)
I20260812 16:41:33.503108 13697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:41:33.507997 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 13697
I20260812 16:41:33.508198 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:41:33.508817 31990 external_mini_cluster.cc:1428] Setting key c639355d90911f2c626e7232d21f83e7
Aug 12 16:41:33 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:33.525279 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:33.505720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:53193 (local address 127.31.61.190:36469)
0812 16:41:33.506168 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:33.506172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:33.507005 (+ 833us) server_negotiation.cc:407] Connection header received
0812 16:41:33.508939 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:33.508945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:33.509030 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:33.509130 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:33.510861 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:33.511704 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:33.513967 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:33.514369 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:33.514514 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:33.518830 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:33.518835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:33.518839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:33.518874 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:33.521952 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:33.522789 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:33.522796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:33.522799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:33.522951 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:33.523380 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:33.523384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:33.523387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:33.523764 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:33.523902 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:33.524636 (+ 734us) server_negotiation.cc:299] Negotiation successful
0812 16:41:33.524926 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260812 16:41:33.526079 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:33.506129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:53193)
0812 16:41:33.506825 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:33.506870 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:33.508638 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:33.509389 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:33.509398 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:33.510042 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:33.510649 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:33.510676 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:33.511879 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:33.511884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:33.513782 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:33.513800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:33.514638 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:33.514645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:33.514937 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:33.515685 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:33.515713 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:33.518548 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:33.522139 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:33.522148 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:33.522164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:33.522644 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:33.523074 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:33.523077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:33.523079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:33.523277 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:33.523890 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:33.523897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:33.524237 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 16:41:33.524510 (+ 273us) client_negotiation.cc:240] Negotiation successful
0812 16:41:33.524932 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":267,"threads_started":1}
I20260812 16:41:33.527661 13825 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36469
I20260812 16:41:33.527988 13825 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:33.528765 13825 heartbeater.cc:511] Master 127.31.61.190:36469 requested a full tablet report, sending...
I20260812 16:41:33.531301 13636 ts_manager.cc:194] Registered new tserver with Master: 4f699173dce3457ba9d34619d8d932e8 (127.31.61.129:35119)
I20260812 16:41:33.532948 13636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:53193
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:41:33.568053 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36469
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:33.907873 13832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:33.908144 13832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:33.908241 13832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:33.915512 13832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:33.915669 13832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:33.915803 13832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:41:33.925690 13832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:33.926952 13832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:33.928323 13832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:33.941903 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:33.941920 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:33.942039 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:33.942546 13832 server_base.cc:1034] running on GCE node
I20260812 16:41:33.943429 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:33.944556 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:33.945789 13832 hybrid_clock.cc:648] HybridClock initialized: now 1786552893945744 us; error 35 us; skew 500 ppm
I20260812 16:41:33.946116 13832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:33 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:33.949491 13832 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:41:33.951058 13832 webserver.cc:460] Webserver started at http://127.31.61.130:34131/ using document root <none> and password file <none>
I20260812 16:41:33.951805 13832 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:33.951930 13832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:33.952230 13832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:33.955228 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "f448b863e37646c7affecd3f8436fc79"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "d770db2d0bc8c17f716b4f4c476cba99"
server_key_iv: "5f5b04ae823c13e6f562ac363942dac7"
server_key_version: "encryptionkey@0"
I20260812 16:41:33.956082 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "f448b863e37646c7affecd3f8436fc79"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "d770db2d0bc8c17f716b4f4c476cba99"
server_key_iv: "5f5b04ae823c13e6f562ac363942dac7"
server_key_version: "encryptionkey@0"
I20260812 16:41:33.961704 13832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:41:33.965530 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:33.967032 13832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:41:33.967200 13832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "f448b863e37646c7affecd3f8436fc79"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "d770db2d0bc8c17f716b4f4c476cba99"
server_key_iv: "5f5b04ae823c13e6f562ac363942dac7"
server_key_version: "encryptionkey@0"
I20260812 16:41:33.967435 13832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:33.985026 13832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:33.988948 13832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:33.989214 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:33.990157 13832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:33.991555 13832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:33.991657 13832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 16:41:33.991777 13832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:33.991830 13832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:34.023573 13832 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:43715
I20260812 16:41:34.023644 13960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:43715 every 8 connection(s)
I20260812 16:41:34.024845 13832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:41:34.032380 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 13832
I20260812 16:41:34.032577 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:41:34.033047 31990 external_mini_cluster.cc:1428] Setting key fd5af10721e2eb555b4165666d4690b3
Aug 12 16:41:34 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:34.046211 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.027406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:41441 (local address 127.31.61.190:36469)
0812 16:41:34.027849 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.027853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.028732 (+ 879us) server_negotiation.cc:407] Connection header received
0812 16:41:34.030522 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.030528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.030618 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.030716 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:34.032504 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.033349 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.035306 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.035875 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.036006 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.040127 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:34.040134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:34.040137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:34.040175 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:34.042939 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:34.043838 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:34.043843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:34.043845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:34.043999 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:34.044457 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:34.044462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:34.044465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:34.044750 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:34.044887 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.045612 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.045838 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":242,"threads_started":1}
I20260812 16:41:34.047235 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.027851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:41441)
0812 16:41:34.028578 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:34.028622 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:34.030249 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:34.030961 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:34.030969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:34.031482 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:34.032231 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.032258 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.033536 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.033541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.035145 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.035158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.036124 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.036131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.036392 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.037139 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:34.037175 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:34.039859 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:34.043092 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:34.043102 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:34.043121 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:34.043722 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:34.044121 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:34.044126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:34.044131 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:34.044331 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:34.044878 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:34.044887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:34.045247 (+ 360us) client_negotiation.cc:769] Sending connection context
0812 16:41:34.045499 (+ 252us) client_negotiation.cc:240] Negotiation successful
0812 16:41:34.046052 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":263,"threads_started":1}
I20260812 16:41:34.048833 13961 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36469
I20260812 16:41:34.049136 13961 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:34.049894 13961 heartbeater.cc:511] Master 127.31.61.190:36469 requested a full tablet report, sending...
I20260812 16:41:34.051401 13636 ts_manager.cc:194] Registered new tserver with Master: f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715)
I20260812 16:41:34.052461 13636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:41441
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:41:34.093168 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36469
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:34.431279 13967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:34.431648 13967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:34.431797 13967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:34.439749 13967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:34.439890 13967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:34.440048 13967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:41:34.450758 13967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:34.452364 13967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:34.453787 13967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:34.466805 13972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:34.466769 13973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:34.466843 13975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:34.468014 13967 server_base.cc:1034] running on GCE node
I20260812 16:41:34.468662 13967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:34.469602 13967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:34.470839 13967 hybrid_clock.cc:648] HybridClock initialized: now 1786552894470771 us; error 44 us; skew 500 ppm
I20260812 16:41:34.471148 13967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:34 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552894, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:34.474447 13967 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:41:34.476083 13967 webserver.cc:460] Webserver started at http://127.31.61.131:44989/ using document root <none> and password file <none>
I20260812 16:41:34.476795 13967 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:34.476907 13967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:34.477170 13967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:34.480378 13967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "50f50261d31d4cd186c1b69e905ba050"
format_stamp: "Formatted at 2026-08-12 16:41:34 on dist-test-slave-f7th"
server_key: "2518398b31673fc7b25fd8f875176e95"
server_key_iv: "ec709db24df070b232c8a50afaae4be3"
server_key_version: "encryptionkey@0"
I20260812 16:41:34.481150 13967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "50f50261d31d4cd186c1b69e905ba050"
format_stamp: "Formatted at 2026-08-12 16:41:34 on dist-test-slave-f7th"
server_key: "2518398b31673fc7b25fd8f875176e95"
server_key_iv: "ec709db24df070b232c8a50afaae4be3"
server_key_version: "encryptionkey@0"
I20260812 16:41:34.486788 13967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 16:41:34.490566 13982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:34.491997 13967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:41:34.492223 13967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "50f50261d31d4cd186c1b69e905ba050"
format_stamp: "Formatted at 2026-08-12 16:41:34 on dist-test-slave-f7th"
server_key: "2518398b31673fc7b25fd8f875176e95"
server_key_iv: "ec709db24df070b232c8a50afaae4be3"
server_key_version: "encryptionkey@0"
I20260812 16:41:34.492442 13967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:34.509325 13967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:34.513239 13967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:34.513478 13967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:34.514370 13967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:34.515724 13967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:34.515797 13967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:34.515898 13967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:34.515942 13967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:34.536582 13825 heartbeater.cc:503] Master 127.31.61.190:36469 was elected leader, sending a full tablet report...
I20260812 16:41:34.548858 13967 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:34351
I20260812 16:41:34.548904 14095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:34351 every 8 connection(s)
I20260812 16:41:34.550182 13967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:41:34.558107 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 13967
I20260812 16:41:34.558277 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:41:34.558748 31990 external_mini_cluster.cc:1428] Setting key 0f3213a11b4d15ed9875f2d25f3d44bf
Aug 12 16:41:34 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552894, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:34.569743 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.552754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:55569 (local address 127.31.61.190:36469)
0812 16:41:34.553214 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.553219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.554191 (+ 972us) server_negotiation.cc:407] Connection header received
0812 16:41:34.556041 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.556047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.556140 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.556253 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:34.557741 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.558589 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.560505 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.560996 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.561128 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.564758 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:34.564764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:34.564767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:34.564801 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:34.567111 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:34.567819 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:34.567823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:34.567825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:34.567943 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:34.568299 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:34.568303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:34.568304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:34.568554 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:34.568667 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.569271 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.569446 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260812 16:41:34.570700 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.553285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:55569)
0812 16:41:34.554039 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:34.554077 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:34.555744 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:34.556464 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:34.556473 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:34.557007 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:34.557561 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.557579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.558756 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.558761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.560357 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.560374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.561274 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.561282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.561588 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.562393 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:34.562416 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:34.564555 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:34.567238 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:34.567245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:34.567258 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:34.567723 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:34.568020 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:34.568023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:34.568025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:34.568218 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:34.568644 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:34.568650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:34.568962 (+ 312us) client_negotiation.cc:769] Sending connection context
0812 16:41:34.569207 (+ 245us) client_negotiation.cc:240] Negotiation successful
0812 16:41:34.569609 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":260,"threads_started":1}
I20260812 16:41:34.572084 14096 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36469
I20260812 16:41:34.572372 14096 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:34.573199 14096 heartbeater.cc:511] Master 127.31.61.190:36469 requested a full tablet report, sending...
I20260812 16:41:34.574689 13635 ts_manager.cc:194] Registered new tserver with Master: 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351)
I20260812 16:41:34.575541 13635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:55569
I20260812 16:41:34.587289 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:41:34.603963 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.591735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.31.61.190:36469)
0812 16:41:34.591924 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.591930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.592314 (+ 384us) server_negotiation.cc:407] Connection header received
0812 16:41:34.592478 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.592482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.592569 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.592652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:34.593989 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.594722 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.596083 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.596556 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.596681 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.599086 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:34.599091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:34.599094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:34.599124 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:34.601513 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:34.602104 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:34.602110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:34.602113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:34.602238 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:34.602566 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:34.602571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:34.602574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:34.602885 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:34.603107 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.603437 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.603682 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 16:41:34.608029 13635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:41:34.614923 13635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:41:34.648607 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.627060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:34351 (local address 127.0.0.1:48328)
0812 16:41:34.640453 (+ 13393us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:34.640470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:34.640595 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:34.641727 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:34.641736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:34.641771 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:34.642221 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.642240 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.643946 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.643951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.646145 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.646156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.647642 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.647648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.647843 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.647893 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 16:41:34.648045 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 16:41:34.648239 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13265,"thread_start_us":12192,"threads_started":1}
I20260812 16:41:34.649645 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.627440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.0.0.1:59712)
0812 16:41:34.641054 (+ 13614us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:34.641070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:34.641206 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:34.642110 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:34.642119 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:34.642145 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:34.642619 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.642632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.644486 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.644492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.646796 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.646811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.648817 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.648823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.648991 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.649022 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 16:41:34.649104 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 16:41:34.649225 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13477,"mutex_wait_us":3,"thread_start_us":13340,"threads_started":1}
I20260812 16:41:34.649775 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.627727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.31.61.131:34351)
0812 16:41:34.637659 (+ 9932us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.637670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.640589 (+ 2919us) server_negotiation.cc:407] Connection header received
0812 16:41:34.640803 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.640813 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.641202 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.641397 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:34.642454 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.643827 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.646302 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.647443 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.647571 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.647843 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.649271 (+ 1428us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.649445 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9725,"thread_start_us":9539,"threads_started":1}
I20260812 16:41:34.650234 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.627002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:43715 (local address 127.0.0.1:53766)
0812 16:41:34.639682 (+ 12680us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:34.639730 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:34.639949 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:34.643381 (+ 3432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:34.643387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:34.643413 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:34.643762 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.643773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.645502 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.645508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.647351 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.647363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.649456 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.649461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.649579 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.649605 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 16:41:34.649686 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 16:41:34.649951 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12378,"thread_start_us":13157,"threads_started":1}
I20260812 16:41:34.650303 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.628179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.31.61.129:35119)
0812 16:41:34.640613 (+ 12434us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.640621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.641325 (+ 704us) server_negotiation.cc:407] Connection header received
0812 16:41:34.641451 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.641458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.641660 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.641860 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:34.642840 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.644299 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.646973 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.648314 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.648442 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.648671 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.649917 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.650062 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12271,"thread_start_us":12101,"threads_started":1}
I20260812 16:41:34.651005 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.627144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.31.61.130:43715)
0812 16:41:34.642806 (+ 15662us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.642814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.642840 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:41:34.642941 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.642948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.643171 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.643362 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:34.643933 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.645340 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.647507 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.649038 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.649156 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.649325 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.650572 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.650748 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15501,"thread_start_us":15402,"threads_started":1}
I20260812 16:41:34.653806 13895 tablet_service.cc:1467] Processing CreateTablet for tablet 85198272dd894868afe5f725dac6db85 (DEFAULT_TABLE table=test-table [id=8f24d196b9f446fa805c089f590a4041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:34.653823 14031 tablet_service.cc:1467] Processing CreateTablet for tablet 85198272dd894868afe5f725dac6db85 (DEFAULT_TABLE table=test-table [id=8f24d196b9f446fa805c089f590a4041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:34.653828 13760 tablet_service.cc:1467] Processing CreateTablet for tablet 85198272dd894868afe5f725dac6db85 (DEFAULT_TABLE table=test-table [id=8f24d196b9f446fa805c089f590a4041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:34.655156 13895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85198272dd894868afe5f725dac6db85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:34.655182 13760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85198272dd894868afe5f725dac6db85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:34.655550 14031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85198272dd894868afe5f725dac6db85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:34.667699 14115 tablet_bootstrap.cc:492] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: Bootstrap starting.
I20260812 16:41:34.669783 14116 tablet_bootstrap.cc:492] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: Bootstrap starting.
I20260812 16:41:34.671278 14117 tablet_bootstrap.cc:492] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: Bootstrap starting.
I20260812 16:41:34.671437 14115 tablet_bootstrap.cc:654] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:34.672866 14115 log.cc:824] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:34.673370 14116 tablet_bootstrap.cc:654] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:34.674320 14117 tablet_bootstrap.cc:654] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:34.674702 14116 log.cc:824] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:34.675503 14117 log.cc:824] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:34.677425 14115 tablet_bootstrap.cc:492] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: No bootstrap required, opened a new log
I20260812 16:41:34.677525 14116 tablet_bootstrap.cc:492] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: No bootstrap required, opened a new log
I20260812 16:41:34.677740 14115 ts_tablet_manager.cc:1397] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 16:41:34.677740 14116 ts_tablet_manager.cc:1397] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 16:41:34.677907 14117 tablet_bootstrap.cc:492] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: No bootstrap required, opened a new log
I20260812 16:41:34.678144 14117 ts_tablet_manager.cc:1397] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 16:41:34.685716 14117 raft_consensus.cc:348] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } }
I20260812 16:41:34.686113 14117 raft_consensus.cc:374] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:34.686220 14117 raft_consensus.cc:729] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f50261d31d4cd186c1b69e905ba050, State: Initialized, Role: FOLLOWER
I20260812 16:41:34.686841 14117 consensus_queue.cc:260] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } }
I20260812 16:41:34.687986 14096 heartbeater.cc:503] Master 127.31.61.190:36469 was elected leader, sending a full tablet report...
I20260812 16:41:34.688570 14117 ts_tablet_manager.cc:1428] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260812 16:41:34.688328 14115 raft_consensus.cc:348] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } }
I20260812 16:41:34.688810 14115 raft_consensus.cc:374] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:34.688920 14115 raft_consensus.cc:729] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f699173dce3457ba9d34619d8d932e8, State: Initialized, Role: FOLLOWER
I20260812 16:41:34.689532 14115 consensus_queue.cc:260] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } }
I20260812 16:41:34.690130 14116 raft_consensus.cc:348] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } }
I20260812 16:41:34.690579 14116 raft_consensus.cc:374] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:34.690655 14116 raft_consensus.cc:729] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f448b863e37646c7affecd3f8436fc79, State: Initialized, Role: FOLLOWER
I20260812 16:41:34.691148 14116 consensus_queue.cc:260] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } }
I20260812 16:41:34.691291 14115 ts_tablet_manager.cc:1428] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260812 16:41:34.692219 13961 heartbeater.cc:503] Master 127.31.61.190:36469 was elected leader, sending a full tablet report...
I20260812 16:41:34.692849 14116 ts_tablet_manager.cc:1428] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260812 16:41:34.724426 14121 raft_consensus.cc:482] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:34.724761 14121 raft_consensus.cc:504] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } }
I20260812 16:41:34.726498 14121 leader_election.cc:288] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715), 4f699173dce3457ba9d34619d8d932e8 (127.31.61.129:35119)
I20260812 16:41:34.733412 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.726682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:43715 (local address 127.31.61.131:43603)
0812 16:41:34.726967 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:34.726986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:34.727118 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:34.727638 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:34.727643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:34.727671 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:34.727867 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.727878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.729401 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.729405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.731357 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.731372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.732823 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.732828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.732963 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.732989 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 16:41:34.733067 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 16:41:34.733165 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":48}
I20260812 16:41:34.733491 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.726992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:43603 (local address 127.31.61.130:43715)
0812 16:41:34.727229 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.727237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.727272 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:41:34.727369 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.727375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.727453 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.727569 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:34.728019 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.729256 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.731555 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.732658 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.732810 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.732887 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.733106 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.733244 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:41:34.734197 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.727040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:60713 (local address 127.31.61.129:35119)
0812 16:41:34.727263 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.727270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.727841 (+ 571us) server_negotiation.cc:407] Connection header received
0812 16:41:34.727974 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.727980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.728063 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.728169 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:34.728643 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.729695 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.732117 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.733298 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.733452 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.733519 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.733828 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.733960 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 16:41:34.734202 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.726866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.31.61.131:60713)
0812 16:41:34.727704 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:34.727725 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:34.727840 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:34.728176 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:34.728180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:34.728205 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:34.728469 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.728477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.729845 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.729850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.731951 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:34.731965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.733507 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.733513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:34.733677 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.733708 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 16:41:34.733789 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 16:41:34.733905 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":1,"thread_start_us":465,"threads_started":1}
I20260812 16:41:34.734750 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85198272dd894868afe5f725dac6db85" candidate_uuid: "50f50261d31d4cd186c1b69e905ba050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f448b863e37646c7affecd3f8436fc79" is_pre_election: true
I20260812 16:41:34.735281 13915 raft_consensus.cc:2454] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f50261d31d4cd186c1b69e905ba050 in term 0.
I20260812 16:41:34.735256 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85198272dd894868afe5f725dac6db85" candidate_uuid: "50f50261d31d4cd186c1b69e905ba050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f699173dce3457ba9d34619d8d932e8" is_pre_election: true
I20260812 16:41:34.735803 13780 raft_consensus.cc:2454] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f50261d31d4cd186c1b69e905ba050 in term 0.
I20260812 16:41:34.735939 13985 leader_election.cc:302] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50f50261d31d4cd186c1b69e905ba050, f448b863e37646c7affecd3f8436fc79; no voters:
I20260812 16:41:34.736330 14121 raft_consensus.cc:2781] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:41:34.736451 14121 raft_consensus.cc:482] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:41:34.736547 14121 raft_consensus.cc:3037] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:34.738896 14121 raft_consensus.cc:504] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } }
I20260812 16:41:34.739501 14121 leader_election.cc:288] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [CANDIDATE]: Term 1 election: Requested vote from peers f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715), 4f699173dce3457ba9d34619d8d932e8 (127.31.61.129:35119)
I20260812 16:41:34.739976 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85198272dd894868afe5f725dac6db85" candidate_uuid: "50f50261d31d4cd186c1b69e905ba050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f448b863e37646c7affecd3f8436fc79"
I20260812 16:41:34.740072 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85198272dd894868afe5f725dac6db85" candidate_uuid: "50f50261d31d4cd186c1b69e905ba050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f699173dce3457ba9d34619d8d932e8"
I20260812 16:41:34.740191 13915 raft_consensus.cc:3037] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:34.740268 13780 raft_consensus.cc:3037] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:34.743428 13780 raft_consensus.cc:2454] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f50261d31d4cd186c1b69e905ba050 in term 1.
I20260812 16:41:34.743428 13915 raft_consensus.cc:2454] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f50261d31d4cd186c1b69e905ba050 in term 1.
I20260812 16:41:34.743844 13985 leader_election.cc:302] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50f50261d31d4cd186c1b69e905ba050, f448b863e37646c7affecd3f8436fc79; no voters:
I20260812 16:41:34.744154 14121 raft_consensus.cc:2781] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:41:34.744571 14121 raft_consensus.cc:686] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 1 LEADER]: Becoming Leader. State: Replica: 50f50261d31d4cd186c1b69e905ba050, State: Running, Role: LEADER
I20260812 16:41:34.745043 14121 consensus_queue.cc:237] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } }
I20260812 16:41:34.750248 13633 catalog_manager.cc:5495] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131). New cstate: current_term: 1 leader_uuid: "50f50261d31d4cd186c1b69e905ba050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } health_report { overall_health: UNKNOWN } } }
W20260812 16:41:34.755666 13826 tablet.cc:2369] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:41:34.776734 13962 tablet.cc:2369] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:41:34.781674 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.776041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.31.61.131:34351)
0812 16:41:34.776237 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.776243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.776298 (+ 55us) server_negotiation.cc:407] Connection header received
0812 16:41:34.776551 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.776555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.776638 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.776771 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:41:34.777185 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.777917 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.779395 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.779839 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.779960 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.780554 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:41:34.781083 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:41:34.781229 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.781332 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.781418 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 16:41:34.790077 13915 raft_consensus.cc:1261] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [term 1 FOLLOWER]: Refusing update from remote peer 50f50261d31d4cd186c1b69e905ba050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:34.790069 13780 raft_consensus.cc:1261] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Refusing update from remote peer 50f50261d31d4cd186c1b69e905ba050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:34.791079 14125 consensus_queue.cc:1035] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:41:34.791400 14121 consensus_queue.cc:1035] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:41:34.801755 14097 tablet.cc:2369] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:41:34.806433 14129 mvcc.cc:205] Tried to move back new op lower bound from 7317720657049571328 to 7317720656879464448. Current Snapshot: MvccSnapshot[applied={T|T < 7317720657049571328}]
I20260812 16:41:34.807813 14128 mvcc.cc:205] Tried to move back new op lower bound from 7317720657049571328 to 7317720656879464448. Current Snapshot: MvccSnapshot[applied={T|T < 7317720657049571328}]
I20260812 16:41:34.819708 13633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:41:34.820039 13633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:41:34.823987 13633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Sending DeleteTablet for 3 replicas of tablet 85198272dd894868afe5f725dac6db85
I20260812 16:41:34.825251 14031 tablet_service.cc:1514] Processing DeleteTablet for tablet 85198272dd894868afe5f725dac6db85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:34 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:48328
I20260812 16:41:34.825519 13760 tablet_service.cc:1514] Processing DeleteTablet for tablet 85198272dd894868afe5f725dac6db85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:34 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:59712
I20260812 16:41:34.825639 13895 tablet_service.cc:1514] Processing DeleteTablet for tablet 85198272dd894868afe5f725dac6db85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:34 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:53766
I20260812 16:41:34.825773 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 13697
I20260812 16:41:34.836570 14141 tablet_replica.cc:330] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: stopping tablet replica
I20260812 16:41:34.837839 14141 raft_consensus.cc:2229] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:41:34.839310 14141 raft_consensus.cc:2258] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:34.840474 14140 tablet_replica.cc:330] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: stopping tablet replica
I20260812 16:41:34.840744 14139 tablet_replica.cc:330] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: stopping tablet replica
I20260812 16:41:34.840931 14140 raft_consensus.cc:2229] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:41:34.841228 14139 raft_consensus.cc:2229] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:41:34.841543 14140 raft_consensus.cc:2258] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:34.842144 14139 raft_consensus.cc:2258] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:34.842854 14141 ts_tablet_manager.cc:1905] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:34.843870 14140 ts_tablet_manager.cc:1905] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:34.844242 14139 ts_tablet_manager.cc:1905] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:34.850088 14141 ts_tablet_manager.cc:1918] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:34.850332 14141 log.cc:1196] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/85198272dd894868afe5f725dac6db85
I20260812 16:41:34.851306 14141 ts_tablet_manager.cc:1939] T 85198272dd894868afe5f725dac6db85 P f448b863e37646c7affecd3f8436fc79: Deleting consensus metadata
I20260812 16:41:34.851301 14140 ts_tablet_manager.cc:1918] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:34.851480 14140 log.cc:1196] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/85198272dd894868afe5f725dac6db85
I20260812 16:41:34.852051 14140 ts_tablet_manager.cc:1939] T 85198272dd894868afe5f725dac6db85 P 4f699173dce3457ba9d34619d8d932e8: Deleting consensus metadata
Aug 12 16:41:34 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:41:34.853715 13622 catalog_manager.cc:4841] TS f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715): tablet 85198272dd894868afe5f725dac6db85 (table test-table [id=8f24d196b9f446fa805c089f590a4041]) successfully deleted
I20260812 16:41:34.854158 13621 catalog_manager.cc:4841] TS 4f699173dce3457ba9d34619d8d932e8 (127.31.61.129:35119): tablet 85198272dd894868afe5f725dac6db85 (table test-table [id=8f24d196b9f446fa805c089f590a4041]) successfully deleted
I20260812 16:41:34.855408 14139 ts_tablet_manager.cc:1918] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:34.855638 14139 log.cc:1196] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/85198272dd894868afe5f725dac6db85
I20260812 16:41:34.856236 14139 ts_tablet_manager.cc:1939] T 85198272dd894868afe5f725dac6db85 P 50f50261d31d4cd186c1b69e905ba050: Deleting consensus metadata
I20260812 16:41:34.858029 13620 catalog_manager.cc:4841] TS 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351): tablet 85198272dd894868afe5f725dac6db85 (table test-table [id=8f24d196b9f446fa805c089f590a4041]) successfully deleted
I20260812 16:41:34.860365 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.827125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.31.61.129:35119)
0812 16:41:34.827316 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.827323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.840082 (+ 12759us) server_negotiation.cc:407] Connection header received
0812 16:41:34.840450 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.840460 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.840545 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.840624 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:34.843829 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.845091 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.847199 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.847901 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.848003 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.854035 (+ 6032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:34.854051 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:34.854069 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:34.854123 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:34.857149 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:34.857976 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:34.857987 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:34.857991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:34.858174 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:34.858567 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:34.858573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:34.858576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:34.859075 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:34.859391 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:34.859704 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 16:41:34.859981 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:41:34.861166 13820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:34.965615 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 13697
I20260812 16:41:34.988138 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 13832
Aug 12 16:41:34 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:41:35.005383 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:34.988932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.31.61.130:43715)
0812 16:41:34.989148 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:34.989153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:34.989174 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:41:34.989331 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:34.989338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:34.989406 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:34.989507 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:34.990914 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.991712 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.993059 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:34.993458 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:34.993573 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:34.998309 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:34.998326 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:34.998343 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:34.998384 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:35.001990 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:35.003004 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:35.003016 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:35.003020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:35.003188 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:35.003662 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:35.003667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:35.003670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:35.004188 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:35.004518 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:35.004762 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 16:41:35.005013 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 16:41:35.006147 13956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:35.096421 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 13832
I20260812 16:41:35.113826 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 13967
Aug 12 16:41:35 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:41:35.131696 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:35.114609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.31.61.131:34351)
0812 16:41:35.114823 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:35.114829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:35.114888 (+ 59us) server_negotiation.cc:407] Connection header received
0812 16:41:35.114957 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:35.114963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:35.115082 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:35.115194 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:35.116553 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:35.117776 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:35.119173 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:35.119900 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:35.120050 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:35.124738 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:35.124761 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:35.124784 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:35.124842 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:35.128769 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:35.129515 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:35.129523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:35.129525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:35.129681 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:35.130066 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:35.130071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:35.130073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:35.130451 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:35.130794 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:35.131071 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 16:41:35.131296 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 16:41:35.132516 14091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:35.254321 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 13967
I20260812 16:41:35.271432 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 13603
I20260812 16:41:35.272643 13666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:35.382862 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 13603
I20260812 16:41:35.401158 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36469
--webserver_interface=127.31.61.190
--webserver_port=45713
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:35.732968 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:35.733268 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:35.733379 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:35.741011 14147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:35.741163 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:35.741230 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:35.741313 14147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:35.741366 14147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:35.751698 14147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=45713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:35.753166 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:35.754549 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:35.777037 14154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:35.777099 14153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:35.777120 14156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:35.777840 14147 server_base.cc:1034] running on GCE node
I20260812 16:41:35.778734 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:35.780439 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:35.781656 14147 hybrid_clock.cc:648] HybridClock initialized: now 1786552895781597 us; error 53 us; skew 500 ppm
I20260812 16:41:35.782073 14147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:35 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552895, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:35.785705 14147 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:41:35.787400 14147 webserver.cc:460] Webserver started at http://127.31.61.190:45713/ using document root <none> and password file <none>
I20260812 16:41:35.788189 14147 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:35.788362 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:35.794402 14147 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 16:41:35.797387 14163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:35.798768 14147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:41:35.798920 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "2d8a13c711934aed9d25ccd53efe8324"
format_stamp: "Formatted at 2026-08-12 16:41:32 on dist-test-slave-f7th"
server_key: "e5a7e0d90a865b9d8146d69cda5d222d"
server_key_iv: "dd2bb86751438974ce293ad409a08e2c"
server_key_version: "encryptionkey@0"
I20260812 16:41:35.800024 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:35.824082 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:35.828470 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:35.828740 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:35.847085 14147 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36469
I20260812 16:41:35.847092 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36469 every 8 connection(s)
I20260812 16:41:35.848637 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:41:35.855291 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 14147
I20260812 16:41:35.855635 14215 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:41:35.856135 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:35119
--local_ip_for_outbound_sockets=127.31.61.129
--tserver_master_addrs=127.31.61.190:36469
--webserver_port=45775
--webserver_interface=127.31.61.129
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:41:35.859040 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Bootstrap starting.
I20260812 16:41:35.882912 14215 log.cc:824] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:35.895816 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:41:35.896346 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Bootstrap complete.
I20260812 16:41:35.905812 14215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:35.906687 14215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8a13c711934aed9d25ccd53efe8324, State: Initialized, Role: FOLLOWER
I20260812 16:41:35.907327 14215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:35.907574 14215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:41:35.907722 14215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:41:35.907909 14215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:41:35.911804 14215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:35.912351 14215 leader_election.cc:302] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d8a13c711934aed9d25ccd53efe8324; no voters:
I20260812 16:41:35.912995 14215 leader_election.cc:288] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 16:41:35.913141 14219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:41:35.914521 14219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 2 LEADER]: Becoming Leader. State: Replica: 2d8a13c711934aed9d25ccd53efe8324, State: Running, Role: LEADER
I20260812 16:41:35.915138 14219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:35.915552 14215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:41:35.917794 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d8a13c711934aed9d25ccd53efe8324. Latest consensus state: current_term: 2 leader_uuid: "2d8a13c711934aed9d25ccd53efe8324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } } }
I20260812 16:41:35.917821 14220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2d8a13c711934aed9d25ccd53efe8324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } } }
I20260812 16:41:35.918426 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:35.918514 14220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:35.929071 14226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:41:35.938319 14226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:41:35.939875 14226 catalog_manager.cc:1259] Loaded cluster ID: 7f6c91398ccc40b6b4117472079e3b4b
I20260812 16:41:35.939981 14226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:41:35.943437 14226 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 16:41:35.959820 14226 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 16:41:35.967211 14226 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:41:35.968609 14226 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Loaded TSK: 0
I20260812 16:41:35.969770 14226 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:41:36.261216 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:36.261528 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:36.261664 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:36.269285 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:36.269424 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:36.269580 14217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:41:36.280087 14217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:35119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=45775
--tserver_master_addrs=127.31.61.190:36469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:36.281493 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:36.282857 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:36.298913 14242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:36.298887 14243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:36.299515 14245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:36.299768 14217 server_base.cc:1034] running on GCE node
I20260812 16:41:36.300428 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:36.301467 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:36.302690 14217 hybrid_clock.cc:648] HybridClock initialized: now 1786552896302634 us; error 43 us; skew 500 ppm
I20260812 16:41:36.303112 14217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:36 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:36.308136 14217 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:41:36.312947 14217 webserver.cc:460] Webserver started at http://127.31.61.129:45775/ using document root <none> and password file <none>
I20260812 16:41:36.313670 14217 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:36.313807 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:36.319412 14217 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:41:36.322180 14252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:36.323568 14217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:41:36.323758 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "4f699173dce3457ba9d34619d8d932e8"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "ec131f77babb350648445818f835a9cd"
server_key_iv: "d1c11db5cf62d38cdf6460b3214c765a"
server_key_version: "encryptionkey@0"
I20260812 16:41:36.324776 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:36.348716 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:36.352924 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:36.353188 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:36.354238 14217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:36.355713 14217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:36.355813 14217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:36.355922 14217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:36.355991 14217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:36.386852 14217 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:35119
I20260812 16:41:36.386924 14367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:35119 every 8 connection(s)
I20260812 16:41:36.388386 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:41:36.397619 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 14217
I20260812 16:41:36.398249 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:43715
--local_ip_for_outbound_sockets=127.31.61.130
--tserver_master_addrs=127.31.61.190:36469
--webserver_port=34131
--webserver_interface=127.31.61.130
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:41:36 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:36.413125 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:36.391481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:38149)
0812 16:41:36.392335 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:36.392395 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:36.394240 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:36.395531 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:36.395538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:36.396097 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:36.396623 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:36.396640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:36.398311 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:36.398316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:36.400181 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:36.400197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:36.401250 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:36.401256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:36.401522 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:36.402222 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:36.402253 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:36.405004 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:36.408938 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:36.408946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:36.408958 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:36.409407 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:36.409968 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:36.409972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:36.409974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:36.410179 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:36.411058 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:36.411064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:36.411367 (+ 303us) client_negotiation.cc:769] Sending connection context
0812 16:41:36.411649 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 16:41:36.412102 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":322,"threads_started":1}
I20260812 16:41:36.413363 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:36.391228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:38149 (local address 127.31.61.190:36469)
0812 16:41:36.391793 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:36.391805 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:36.392558 (+ 753us) server_negotiation.cc:407] Connection header received
0812 16:41:36.394740 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:36.394752 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:36.395068 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:36.395337 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:36.396815 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:36.398134 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:36.400357 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:36.401007 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:36.401174 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:36.405317 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:36.405325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:36.405340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:36.405381 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:36.408761 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:36.409584 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:36.409593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:36.409600 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:36.409810 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:36.410295 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:36.410299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:36.410302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:36.410917 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:36.411103 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:36.411881 (+ 778us) server_negotiation.cc:299] Negotiation successful
0812 16:41:36.412280 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":244,"threads_started":1}
I20260812 16:41:36.416049 14368 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36469
I20260812 16:41:36.416365 14368 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:36.417234 14368 heartbeater.cc:511] Master 127.31.61.190:36469 requested a full tablet report, sending...
I20260812 16:41:36.419430 14180 ts_manager.cc:194] Registered new tserver with Master: 4f699173dce3457ba9d34619d8d932e8 (127.31.61.129:35119)
I20260812 16:41:36.420912 14180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:38149
W20260812 16:41:36.728374 14372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:36.728650 14372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:36.728785 14372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:36.736061 14372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:36.736196 14372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:36.736351 14372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:41:36.746119 14372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:43715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=34131
--tserver_master_addrs=127.31.61.190:36469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:36.747418 14372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:36.748788 14372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:36.761920 14377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:36.761981 14380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:36.761991 14378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:36.762691 14372 server_base.cc:1034] running on GCE node
I20260812 16:41:36.763350 14372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:36.764336 14372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:36.765574 14372 hybrid_clock.cc:648] HybridClock initialized: now 1786552896765503 us; error 50 us; skew 500 ppm
I20260812 16:41:36.765884 14372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:36 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:36.769017 14372 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:41:36.770744 14372 webserver.cc:460] Webserver started at http://127.31.61.130:34131/ using document root <none> and password file <none>
I20260812 16:41:36.771414 14372 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:36.771533 14372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:36.776919 14372 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:41:36.779879 14387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:36.781489 14372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:41:36.781643 14372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "f448b863e37646c7affecd3f8436fc79"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "d770db2d0bc8c17f716b4f4c476cba99"
server_key_iv: "5f5b04ae823c13e6f562ac363942dac7"
server_key_version: "encryptionkey@0"
I20260812 16:41:36.782629 14372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:36.797518 14372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:36.801940 14372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:36.802196 14372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:36.803236 14372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:36.804708 14372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:36.804796 14372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:36.804885 14372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:36.804962 14372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:36.836012 14372 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:43715
I20260812 16:41:36.836097 14500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:43715 every 8 connection(s)
I20260812 16:41:36.837423 14372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:41:36.847043 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 14372
I20260812 16:41:36.847826 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:34351
--local_ip_for_outbound_sockets=127.31.61.131
--tserver_master_addrs=127.31.61.190:36469
--webserver_port=44989
--webserver_interface=127.31.61.131
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:41:36 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552896, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:36.857362 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:36.839901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:46473 (local address 127.31.61.190:36469)
0812 16:41:36.840122 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:36.840128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:36.841295 (+ 1167us) server_negotiation.cc:407] Connection header received
0812 16:41:36.843097 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:36.843103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:36.843186 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:36.843272 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:36.844984 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:36.845707 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:36.847418 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:36.847904 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:36.848045 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:36.851582 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:36.851587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:36.851590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:36.851659 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:36.854178 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:36.854913 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:36.854917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:36.854919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:36.855046 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:36.855470 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:36.855474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:36.855476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:36.855828 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:36.856002 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:36.856695 (+ 693us) server_negotiation.cc:299] Negotiation successful
0812 16:41:36.857029 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 16:41:36.858502 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:36.840382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:46473)
0812 16:41:36.841129 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:36.841173 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:36.842820 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:36.843522 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:36.843531 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:36.844097 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:36.844791 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:36.844811 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:36.845859 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:36.845863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:36.847253 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:36.847271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:36.848306 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:36.848311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:36.848524 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:36.849193 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:36.849217 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:36.851349 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:36.854319 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:36.854329 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:36.854347 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:36.854783 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:36.855152 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:36.855158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:36.855161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:36.855353 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:36.855955 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:36.855962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:36.856314 (+ 352us) client_negotiation.cc:769] Sending connection context
0812 16:41:36.856572 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 16:41:36.857119 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":279,"threads_started":1}
I20260812 16:41:36.860050 14501 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36469
I20260812 16:41:36.860404 14501 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:36.861315 14501 heartbeater.cc:511] Master 127.31.61.190:36469 requested a full tablet report, sending...
I20260812 16:41:36.862998 14180 ts_manager.cc:194] Registered new tserver with Master: f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715)
I20260812 16:41:36.864094 14180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:46473
W20260812 16:41:37.181319 14504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:37.181607 14504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:37.181727 14504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:37.189236 14504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:37.189350 14504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:37.189502 14504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:41:37.199579 14504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:34351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=44989
--tserver_master_addrs=127.31.61.190:36469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:37.200937 14504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:37.202293 14504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:37.214932 14509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:37.215091 14510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:37.215063 14513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:37.215938 14504 server_base.cc:1034] running on GCE node
I20260812 16:41:37.216536 14504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:37.217526 14504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:37.218770 14504 hybrid_clock.cc:648] HybridClock initialized: now 1786552897218717 us; error 41 us; skew 500 ppm
I20260812 16:41:37.219108 14504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:37 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552897, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:37.222383 14504 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:41:37.224032 14504 webserver.cc:460] Webserver started at http://127.31.61.131:44989/ using document root <none> and password file <none>
I20260812 16:41:37.224730 14504 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:37.224887 14504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:37.230619 14504 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:41:37.233641 14520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:37.235075 14504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:41:37.235242 14504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "50f50261d31d4cd186c1b69e905ba050"
format_stamp: "Formatted at 2026-08-12 16:41:34 on dist-test-slave-f7th"
server_key: "2518398b31673fc7b25fd8f875176e95"
server_key_iv: "ec709db24df070b232c8a50afaae4be3"
server_key_version: "encryptionkey@0"
I20260812 16:41:37.236299 14504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:37.254940 14504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:37.259301 14504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:37.259563 14504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:37.260654 14504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:37.262068 14504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:37.262171 14504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:37.262286 14504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:37.262354 14504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:37.295276 14504 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:34351
I20260812 16:41:37.295346 14633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:34351 every 8 connection(s)
I20260812 16:41:37.296782 14504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:41:37.306660 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 14504
Aug 12 16:41:37 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552897, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:37.317907 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.299309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39147 (local address 127.31.61.190:36469)
0812 16:41:37.299551 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.299556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.300697 (+ 1141us) server_negotiation.cc:407] Connection header received
0812 16:41:37.302536 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.302542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.302633 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.302718 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:37.304390 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.305173 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.306732 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.307164 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.307275 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.311212 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:37.311218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:37.311221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:37.311253 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:37.314593 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.315346 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.315352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.315356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.315509 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.315982 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.315987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.315990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.316371 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:37.316523 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.317285 (+ 762us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.317528 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 16:41:37.318976 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.299829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:39147)
0812 16:41:37.300534 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.300579 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:37.302237 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:37.302974 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:37.302983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:37.303560 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:37.304219 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.304239 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.305331 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.305335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.306589 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.306602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.307399 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.307407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.307688 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.308429 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:37.308468 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:37.310983 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:37.314734 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:37.314743 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:37.314758 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:37.315213 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:37.315649 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:37.315652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:37.315654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:37.315868 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:37.316494 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:37.316501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:37.316858 (+ 357us) client_negotiation.cc:769] Sending connection context
0812 16:41:37.317134 (+ 276us) client_negotiation.cc:240] Negotiation successful
0812 16:41:37.317715 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":264,"threads_started":1}
I20260812 16:41:37.320467 14634 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36469
I20260812 16:41:37.320747 14634 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:37.321486 14634 heartbeater.cc:511] Master 127.31.61.190:36469 requested a full tablet report, sending...
I20260812 16:41:37.322235 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.307573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.31.61.190:36469)
0812 16:41:37.308094 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.308099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.308429 (+ 330us) server_negotiation.cc:407] Connection header received
0812 16:41:37.308633 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.308638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.308712 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.308818 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:37.310213 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.311256 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.313176 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.313712 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.313818 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.316809 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:37.316814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:37.316817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:37.316845 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:37.319630 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.320338 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.320342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.320344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.320453 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.320821 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.320826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.320828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.321119 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:37.321351 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.321652 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.321866 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":247,"threads_started":1}
I20260812 16:41:37.322950 14180 ts_manager.cc:194] Registered new tserver with Master: 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351)
I20260812 16:41:37.323804 14180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:39147
I20260812 16:41:37.324921 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:41:37.331683 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.325876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.31.61.190:36469)
0812 16:41:37.326025 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.326030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.326414 (+ 384us) server_negotiation.cc:407] Connection header received
0812 16:41:37.326625 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.326629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.326749 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.326879 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:41:37.327288 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.328003 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.329559 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.330050 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.330165 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.330671 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:41:37.331162 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:41:37.331279 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.331359 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.331441 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 16:41:37.332674 14179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:41:37.340287 14179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:41:37.364261 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.352290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.0.0.1:59740)
0812 16:41:37.353970 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.354029 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:37.354226 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:37.356770 (+ 2544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:37.356781 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:37.356816 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:37.357270 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.357288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.359025 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.359038 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.361672 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.361692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.363230 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.363237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.363424 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.363471 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 16:41:37.363592 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 16:41:37.363815 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"mutex_wait_us":58,"thread_start_us":4347,"threads_started":1}
I20260812 16:41:37.364959 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.352205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.31.61.129:35119)
0812 16:41:37.356066 (+ 3861us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.356075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.356106 (+ 31us) server_negotiation.cc:407] Connection header received
0812 16:41:37.356218 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.356227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.356488 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.356702 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:37.357551 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.358858 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.361863 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.363005 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.363147 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.363367 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.363681 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.363858 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3681,"thread_start_us":3566,"threads_started":1}
I20260812 16:41:37.365563 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.352657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.31.61.130:43715)
0812 16:41:37.353278 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.353287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.357059 (+ 3772us) server_negotiation.cc:407] Connection header received
0812 16:41:37.357234 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.357241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.357471 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.357654 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:37.358530 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.359902 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.362291 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.363779 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.363917 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.364109 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.365089 (+ 980us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.365264 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":284,"threads_started":1}
I20260812 16:41:37.365772 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.352696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:43715 (local address 127.0.0.1:53784)
0812 16:41:37.356924 (+ 4228us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.356943 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:37.357067 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:37.357931 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:37.357935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:37.357958 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:37.358314 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.358327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.360042 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.360048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.362112 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.362130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.364285 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.364291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.364765 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.364800 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:41:37.364881 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 16:41:37.365020 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4084,"mutex_wait_us":67,"thread_start_us":5561,"threads_started":1}
I20260812 16:41:37.366205 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.353534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.31.61.131:34351)
0812 16:41:37.356426 (+ 2892us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.356433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.357692 (+ 1259us) server_negotiation.cc:407] Connection header received
0812 16:41:37.357970 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.357977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.358180 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.358344 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:37.359338 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.360952 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.363390 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.364787 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.364890 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.365096 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.365736 (+ 640us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.365900 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2733,"thread_start_us":2593,"threads_started":1}
I20260812 16:41:37.366957 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.353121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:34351 (local address 127.0.0.1:48352)
0812 16:41:37.357583 (+ 4462us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.357606 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:37.357792 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:37.358661 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:37.358666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:37.358689 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:37.359112 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.359125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.361151 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.361156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.363191 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.363211 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.365296 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.365302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.365475 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.365505 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 16:41:37.366467 (+ 962us) client_negotiation.cc:240] Negotiation successful
0812 16:41:37.366621 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4337,"mutex_wait_us":39,"thread_start_us":425,"threads_started":1}
I20260812 16:41:37.369500 14435 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9e6a766ddf45a8beb5a618ba638cf0 (DEFAULT_TABLE table=test-table [id=a1957a8aa8794570a62ccd8309ce2ba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:37.369508 14301 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9e6a766ddf45a8beb5a618ba638cf0 (DEFAULT_TABLE table=test-table [id=a1957a8aa8794570a62ccd8309ce2ba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:37.369859 14568 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9e6a766ddf45a8beb5a618ba638cf0 (DEFAULT_TABLE table=test-table [id=a1957a8aa8794570a62ccd8309ce2ba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:37.370939 14568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9e6a766ddf45a8beb5a618ba638cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:37.370939 14435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9e6a766ddf45a8beb5a618ba638cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:37.370932 14301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9e6a766ddf45a8beb5a618ba638cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:37.384413 14650 tablet_bootstrap.cc:492] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: Bootstrap starting.
I20260812 16:41:37.386569 14651 tablet_bootstrap.cc:492] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Bootstrap starting.
I20260812 16:41:37.388190 14650 tablet_bootstrap.cc:654] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:37.389436 14650 log.cc:824] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:37.389972 14652 tablet_bootstrap.cc:492] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: Bootstrap starting.
I20260812 16:41:37.393577 14652 tablet_bootstrap.cc:654] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:37.394263 14651 tablet_bootstrap.cc:654] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:37.394860 14652 log.cc:824] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:37.395553 14651 log.cc:824] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:37.400662 14650 tablet_bootstrap.cc:492] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: No bootstrap required, opened a new log
I20260812 16:41:37.400775 14652 tablet_bootstrap.cc:492] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: No bootstrap required, opened a new log
I20260812 16:41:37.400936 14650 ts_tablet_manager.cc:1397] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260812 16:41:37.401033 14652 ts_tablet_manager.cc:1397] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260812 16:41:37.406190 14651 tablet_bootstrap.cc:492] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: No bootstrap required, opened a new log
I20260812 16:41:37.406500 14651 ts_tablet_manager.cc:1397] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20260812 16:41:37.408475 14650 raft_consensus.cc:348] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:37.408830 14650 raft_consensus.cc:374] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:37.408942 14650 raft_consensus.cc:729] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f50261d31d4cd186c1b69e905ba050, State: Initialized, Role: FOLLOWER
I20260812 16:41:37.409664 14650 consensus_queue.cc:260] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:37.410472 14652 raft_consensus.cc:348] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:37.410737 14634 heartbeater.cc:503] Master 127.31.61.190:36469 was elected leader, sending a full tablet report...
I20260812 16:41:37.410892 14652 raft_consensus.cc:374] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:37.410981 14652 raft_consensus.cc:729] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f448b863e37646c7affecd3f8436fc79, State: Initialized, Role: FOLLOWER
I20260812 16:41:37.411474 14650 ts_tablet_manager.cc:1428] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260812 16:41:37.411666 14652 consensus_queue.cc:260] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:37.414304 14501 heartbeater.cc:503] Master 127.31.61.190:36469 was elected leader, sending a full tablet report...
I20260812 16:41:37.414870 14652 ts_tablet_manager.cc:1428] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260812 16:41:37.416939 14651 raft_consensus.cc:348] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:37.417431 14651 raft_consensus.cc:374] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:37.417531 14651 raft_consensus.cc:729] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f699173dce3457ba9d34619d8d932e8, State: Initialized, Role: FOLLOWER
I20260812 16:41:37.418040 14651 consensus_queue.cc:260] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:37.419142 14368 heartbeater.cc:503] Master 127.31.61.190:36469 was elected leader, sending a full tablet report...
I20260812 16:41:37.419864 14651 ts_tablet_manager.cc:1428] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Time spent starting tablet: real 0.013s user 0.005s sys 0.010s
I20260812 16:41:37.430872 14657 raft_consensus.cc:482] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:37.431246 14657 raft_consensus.cc:504] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:37.432968 14657 leader_election.cc:288] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f699173dce3457ba9d34619d8d932e8 (127.31.61.129:35119), 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351)
I20260812 16:41:37.439580 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.433557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:45291 (local address 127.31.61.131:34351)
0812 16:41:37.433774 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.433781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.434324 (+ 543us) server_negotiation.cc:407] Connection header received
0812 16:41:37.434517 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.434521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.434588 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.434664 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:37.435237 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.436015 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.437892 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.438667 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.438813 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.438887 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.439207 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.439333 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 16:41:37.439589 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.433402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:42273 (local address 127.31.61.129:35119)
0812 16:41:37.433608 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.433615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.434408 (+ 793us) server_negotiation.cc:407] Connection header received
0812 16:41:37.434578 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.434582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.434641 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.434717 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:37.435329 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.436079 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.437882 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.438655 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.438809 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.438885 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.439197 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.439317 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:41:37.439756 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.433373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:34351 (local address 127.31.61.130:45291)
0812 16:41:37.434200 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.434220 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:37.434382 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:37.434696 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:37.434700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:37.434723 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:37.435095 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.435105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.436139 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.436145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.437740 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.437753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.438921 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.438928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.439065 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.439088 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 16:41:37.439167 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 16:41:37.439293 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":56,"thread_start_us":401,"threads_started":1}
I20260812 16:41:37.439963 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.433464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.31.61.130:42273)
0812 16:41:37.434302 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.434317 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:37.434452 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:37.434742 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:37.434746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:37.434769 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:37.435190 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.435200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.436211 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.436215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.437730 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:37.437744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.438885 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.438892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:37.439051 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.439075 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 16:41:37.439156 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 16:41:37.439291 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":46,"thread_start_us":514,"threads_started":1}
I20260812 16:41:37.440886 14588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9e6a766ddf45a8beb5a618ba638cf0" candidate_uuid: "f448b863e37646c7affecd3f8436fc79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50f50261d31d4cd186c1b69e905ba050" is_pre_election: true
I20260812 16:41:37.440919 14321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9e6a766ddf45a8beb5a618ba638cf0" candidate_uuid: "f448b863e37646c7affecd3f8436fc79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f699173dce3457ba9d34619d8d932e8" is_pre_election: true
I20260812 16:41:37.441431 14321 raft_consensus.cc:2454] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f448b863e37646c7affecd3f8436fc79 in term 0.
I20260812 16:41:37.441421 14588 raft_consensus.cc:2454] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f448b863e37646c7affecd3f8436fc79 in term 0.
I20260812 16:41:37.442137 14389 leader_election.cc:302] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f699173dce3457ba9d34619d8d932e8, f448b863e37646c7affecd3f8436fc79; no voters:
I20260812 16:41:37.442564 14657 raft_consensus.cc:2781] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:41:37.442731 14657 raft_consensus.cc:482] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:41:37.442833 14657 raft_consensus.cc:3037] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:37.445040 14657 raft_consensus.cc:504] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:37.445642 14657 leader_election.cc:288] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [CANDIDATE]: Term 1 election: Requested vote from peers 4f699173dce3457ba9d34619d8d932e8 (127.31.61.129:35119), 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351)
I20260812 16:41:37.446220 14588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9e6a766ddf45a8beb5a618ba638cf0" candidate_uuid: "f448b863e37646c7affecd3f8436fc79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50f50261d31d4cd186c1b69e905ba050"
I20260812 16:41:37.446224 14321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9e6a766ddf45a8beb5a618ba638cf0" candidate_uuid: "f448b863e37646c7affecd3f8436fc79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f699173dce3457ba9d34619d8d932e8"
I20260812 16:41:37.446441 14588 raft_consensus.cc:3037] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:37.446441 14321 raft_consensus.cc:3037] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:37.449604 14588 raft_consensus.cc:2454] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f448b863e37646c7affecd3f8436fc79 in term 1.
I20260812 16:41:37.449604 14321 raft_consensus.cc:2454] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f448b863e37646c7affecd3f8436fc79 in term 1.
I20260812 16:41:37.450053 14389 leader_election.cc:302] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f699173dce3457ba9d34619d8d932e8, f448b863e37646c7affecd3f8436fc79; no voters:
I20260812 16:41:37.450362 14657 raft_consensus.cc:2781] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:41:37.450817 14657 raft_consensus.cc:686] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 1 LEADER]: Becoming Leader. State: Replica: f448b863e37646c7affecd3f8436fc79, State: Running, Role: LEADER
I20260812 16:41:37.451273 14657 consensus_queue.cc:237] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:37.456033 14180 catalog_manager.cc:5495] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 reported cstate change: term changed from 0 to 1, leader changed from <none> to f448b863e37646c7affecd3f8436fc79 (127.31.61.130). New cstate: current_term: 1 leader_uuid: "f448b863e37646c7affecd3f8436fc79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } health_report { overall_health: UNKNOWN } } }
I20260812 16:41:37.474330 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.468953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.31.61.130:43715)
0812 16:41:37.469162 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.469167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.469186 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:41:37.469322 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.469325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.469411 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.469538 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:41:37.469837 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.470573 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.472240 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.472691 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.472790 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.473261 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:41:37.473780 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:41:37.473885 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.473995 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.474066 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:41:37.482733 14588 raft_consensus.cc:1261] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [term 1 FOLLOWER]: Refusing update from remote peer f448b863e37646c7affecd3f8436fc79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:37.482750 14321 raft_consensus.cc:1261] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Refusing update from remote peer f448b863e37646c7affecd3f8436fc79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:37.483701 14661 consensus_queue.cc:1035] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:41:37.484112 14657 consensus_queue.cc:1035] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:41:37.497428 14667 mvcc.cc:205] Tried to move back new op lower bound from 7317720668078665728 to 7317720667964280832. Current Snapshot: MvccSnapshot[applied={T|T < 7317720668078665728}]
I20260812 16:41:37.511986 14179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:41:37.512302 14179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:41:37.516196 14179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Sending DeleteTablet for 3 replicas of tablet 8b9e6a766ddf45a8beb5a618ba638cf0
I20260812 16:41:37.517449 14435 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9e6a766ddf45a8beb5a618ba638cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:37 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:53784
I20260812 16:41:37.517759 14568 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9e6a766ddf45a8beb5a618ba638cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:37 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:48352
I20260812 16:41:37.517809 14301 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9e6a766ddf45a8beb5a618ba638cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:37 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:59740
I20260812 16:41:37.519055 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 14217
I20260812 16:41:37.530166 14677 tablet_replica.cc:330] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: stopping tablet replica
I20260812 16:41:37.530670 14677 raft_consensus.cc:2229] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:41:37.531225 14677 raft_consensus.cc:2258] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:37.535127 14677 ts_tablet_manager.cc:1905] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:37.537578 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.519893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.31.61.129:35119)
0812 16:41:37.520108 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.520114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.520133 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:41:37.520233 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.520237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.520314 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.520421 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:37.522220 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.523372 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.525414 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.526048 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.526188 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.529987 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:37.530007 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:37.530027 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:37.530074 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:37.533904 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.534790 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.534801 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.534805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.534984 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.535509 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.535513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.535517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.536068 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:37.536369 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.536770 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.537029 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 16:41:37.538302 14362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:37.538707 14676 tablet_replica.cc:330] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: stopping tablet replica
I20260812 16:41:37.539330 14676 raft_consensus.cc:2229] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:41:37.540278 14676 raft_consensus.cc:2258] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:37.545557 14676 ts_tablet_manager.cc:1905] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:37.551743 14677 ts_tablet_manager.cc:1918] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:37.557147 14677 log.cc:1196] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/8b9e6a766ddf45a8beb5a618ba638cf0
I20260812 16:41:37.558318 14677 ts_tablet_manager.cc:1939] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 50f50261d31d4cd186c1b69e905ba050: Deleting consensus metadata
I20260812 16:41:37.560554 14676 ts_tablet_manager.cc:1918] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:37.560775 14164 catalog_manager.cc:4841] TS 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351): tablet 8b9e6a766ddf45a8beb5a618ba638cf0 (table test-table [id=a1957a8aa8794570a62ccd8309ce2ba9]) successfully deleted
I20260812 16:41:37.560869 14676 log.cc:1196] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/8b9e6a766ddf45a8beb5a618ba638cf0
I20260812 16:41:37.561642 14676 ts_tablet_manager.cc:1939] T 8b9e6a766ddf45a8beb5a618ba638cf0 P f448b863e37646c7affecd3f8436fc79: Deleting consensus metadata
I20260812 16:41:37.563578 14166 catalog_manager.cc:4841] TS f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715): tablet 8b9e6a766ddf45a8beb5a618ba638cf0 (table test-table [id=a1957a8aa8794570a62ccd8309ce2ba9]) successfully deleted
I20260812 16:41:37.649407 14678 tablet_replica.cc:330] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: stopping tablet replica
I20260812 16:41:37.649682 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 14217
W20260812 16:41:37.666602 14165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.61.129:35119 (error 108)
I20260812 16:41:37.667321 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 14372
I20260812 16:41:37.668812 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.668304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.0.0.1:59762)
0812 16:41:37.668509 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.668633 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 16:41:37.669075 14165 catalog_manager.cc:4584] TS 4f699173dce3457ba9d34619d8d932e8 (127.31.61.129:35119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b9e6a766ddf45a8beb5a618ba638cf0: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35119: connect: Connection refused (error 111)
I20260812 16:41:37.683328 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.668709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.31.61.130:43715)
0812 16:41:37.668871 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.668876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.668896 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:41:37.668962 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.668967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.669073 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.669164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:37.670827 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.671689 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.673061 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.673516 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.673636 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.676456 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:37.676472 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:37.676488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:37.676536 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:37.680077 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.680861 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.680870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.680872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.681034 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.681453 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.681457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.681459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.681879 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:37.682237 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.682697 (+ 460us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.682978 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 16:41:37.684214 14496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:37.723145 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.722617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.0.0.1:59778)
0812 16:41:37.722855 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.722977 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 16:41:37.724249 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.723839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.0.0.1:59790)
0812 16:41:37.724043 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.724136 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 16:41:37.784775 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.784233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.0.0.1:59800)
0812 16:41:37.784499 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.784603 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 16:41:37.785970 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.785487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.0.0.1:59812)
0812 16:41:37.785704 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.785783 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 16:41:37.800515 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 14372
I20260812 16:41:37.817364 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 14504
I20260812 16:41:37.836805 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.818278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.31.61.131:34351)
0812 16:41:37.818585 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:37.818590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:37.818609 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:41:37.818667 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:37.818672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:37.818755 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:37.818873 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:37.820318 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.821112 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.822563 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:37.823116 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:37.823225 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:37.830901 (+ 7676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:37.830922 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:37.830946 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:37.830997 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:37.833907 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.834700 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.834707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.834710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.834866 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:37.835306 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:37.835311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:37.835313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:37.835783 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:37.836015 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:37.836345 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 16:41:37.836530 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 16:41:37.837587 14629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:37.866700 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.866140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.0.0.1:59826)
0812 16:41:37.866388 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.866514 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 16:41:37.868073 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:37.867574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:35119 (local address 127.0.0.1:59828)
0812 16:41:37.867839 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:37.867930 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:35119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260812 16:41:37.928129 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 14504
I20260812 16:41:37.942788 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 14147
I20260812 16:41:37.943971 14210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:38.058512 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 14147
I20260812 16:41:38.077993 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36469
--webserver_interface=127.31.61.190
--webserver_port=45713
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:38.426734 14684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:38.427039 14684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:38.427130 14684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:38.434840 14684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:38.435072 14684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:38.435161 14684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:38.435220 14684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:38.435266 14684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:38.446962 14684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=45713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:38.448472 14684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:38.449779 14684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:38.461759 14689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:38.461781 14690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:38.461714 14692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:38.462582 14684 server_base.cc:1034] running on GCE node
I20260812 16:41:38.463411 14684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:38.464946 14684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:38.466183 14684 hybrid_clock.cc:648] HybridClock initialized: now 1786552898466139 us; error 51 us; skew 500 ppm
I20260812 16:41:38.466559 14684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:38 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:38.469900 14684 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:41:38.471506 14684 webserver.cc:460] Webserver started at http://127.31.61.190:45713/ using document root <none> and password file <none>
I20260812 16:41:38.472244 14684 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:38.472368 14684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:38.478029 14684 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:41:38.481081 14699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:38.482481 14684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:41:38.482654 14684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "2d8a13c711934aed9d25ccd53efe8324"
format_stamp: "Formatted at 2026-08-12 16:41:32 on dist-test-slave-f7th"
server_key: "e5a7e0d90a865b9d8146d69cda5d222d"
server_key_iv: "dd2bb86751438974ce293ad409a08e2c"
server_key_version: "encryptionkey@0"
I20260812 16:41:38.483755 14684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:38.513317 14684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:38.517432 14684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:38.517709 14684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:38.535792 14684 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36469
I20260812 16:41:38.535811 14751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36469 every 8 connection(s)
I20260812 16:41:38.537281 14684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:41:38.543457 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 14684
I20260812 16:41:38.544417 14752 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:41:38.544515 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:35119
--local_ip_for_outbound_sockets=127.31.61.129
--tserver_master_addrs=127.31.61.190:36469
--webserver_port=45775
--webserver_interface=127.31.61.129
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:41:38.548878 14752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Bootstrap starting.
I20260812 16:41:38.570345 14752 log.cc:824] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:38.586784 14752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:41:38.587400 14752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324: Bootstrap complete.
I20260812 16:41:38.597154 14752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:38.598142 14752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8a13c711934aed9d25ccd53efe8324, State: Initialized, Role: FOLLOWER
I20260812 16:41:38.598796 14752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:38.599030 14752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:41:38.599145 14752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:41:38.599306 14752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 2 FOLLOWER]: Advancing to term 3
I20260812 16:41:38.603360 14752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:38.603896 14752 leader_election.cc:302] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d8a13c711934aed9d25ccd53efe8324; no voters:
I20260812 16:41:38.604570 14752 leader_election.cc:288] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 16:41:38.604713 14756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 16:41:38.606825 14752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:41:38.607923 14756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [term 3 LEADER]: Becoming Leader. State: Replica: 2d8a13c711934aed9d25ccd53efe8324, State: Running, Role: LEADER
I20260812 16:41:38.608623 14756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } }
I20260812 16:41:38.611768 14757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2d8a13c711934aed9d25ccd53efe8324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } } }
I20260812 16:41:38.612102 14757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:38.611478 14758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d8a13c711934aed9d25ccd53efe8324. Latest consensus state: current_term: 3 leader_uuid: "2d8a13c711934aed9d25ccd53efe8324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8a13c711934aed9d25ccd53efe8324" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36469 } } }
I20260812 16:41:38.612798 14758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8a13c711934aed9d25ccd53efe8324 [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:38.623915 14767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:41:38.633515 14767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:41:38.635293 14767 catalog_manager.cc:1259] Loaded cluster ID: 7f6c91398ccc40b6b4117472079e3b4b
I20260812 16:41:38.635474 14767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 16:41:38.637791 14767 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7a134be9ad google::LogMessage::Fail() at ??:0
@ 0x7f7a134c188a google::LogMessage::SendToLog() at ??:0
@ 0x7f7a134be4de google::LogMessage::Flush() at ??:0
@ 0x7f7a134bfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7a28ef5512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7a28ef2976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7a28f7a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7a28f7a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7a1610a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7a1610d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7a1610d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7a160e8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7a193c26db start_thread at ??:0
@ 0x7f7a1235861f clone at ??:0
W20260812 16:41:38.920728 14754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:38.921046 14754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:38.921200 14754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:38.928535 14754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:38.928660 14754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:38.928809 14754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:41:38.938681 14754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:35119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=45775
--tserver_master_addrs=127.31.61.190:36469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:38.940233 14754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:38.941589 14754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:38.954883 14795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:38.955236 14754 server_base.cc:1034] running on GCE node
W20260812 16:41:38.954855 14798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:38.954826 14796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:38.956177 14754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:38.957108 14754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:38.958310 14754 hybrid_clock.cc:648] HybridClock initialized: now 1786552898958253 us; error 41 us; skew 500 ppm
I20260812 16:41:38.958606 14754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:38 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:38.961799 14754 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:41:38.963313 14754 webserver.cc:460] Webserver started at http://127.31.61.129:45775/ using document root <none> and password file <none>
I20260812 16:41:38.964062 14754 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:38.964170 14754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:38.970191 14754 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 16:41:38.973446 14805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:38.974998 14754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:41:38.975153 14754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "4f699173dce3457ba9d34619d8d932e8"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "ec131f77babb350648445818f835a9cd"
server_key_iv: "d1c11db5cf62d38cdf6460b3214c765a"
server_key_version: "encryptionkey@0"
I20260812 16:41:38.976213 14754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:38.998582 14754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:39.003096 14754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:39.003376 14754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:39.004436 14754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:39.006424 14812 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 16:41:39.012061 14754 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 16:41:39.012207 14754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260812 16:41:39.012319 14754 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 16:41:39.015336 14754 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 16:41:39.015416 14754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260812 16:41:39.015785 14812 tablet_bootstrap.cc:492] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Bootstrap starting.
I20260812 16:41:39.041882 14812 log.cc:824] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:39.051039 14754 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:35119
I20260812 16:41:39.051050 14920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:35119 every 8 connection(s)
I20260812 16:41:39.052474 14754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:41:39.055308 14812 tablet_bootstrap.cc:492] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:41:39.055912 14812 tablet_bootstrap.cc:492] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Bootstrap complete.
I20260812 16:41:39.057045 14812 ts_tablet_manager.cc:1397] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.015s
I20260812 16:41:39.057358 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.055190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:45037)
0812 16:41:39.056117 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.056438 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":430,"threads_started":1}
W20260812 16:41:39.058080 14921 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:36469 (0 consecutive failures): Network error: Failed to ping master at 127.31.61.190:36469: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
I20260812 16:41:39.059123 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.058725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:43327)
0812 16:41:39.058903 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.058988 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 16:41:39.060186 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.059772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:51323)
0812 16:41:39.059933 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.060037 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 16:41:39.060456 14921 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 16:41:39.063005 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 14754
I20260812 16:41:39.063681 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:43715
--local_ip_for_outbound_sockets=127.31.61.130
--tserver_master_addrs=127.31.61.190:36469
--webserver_port=34131
--webserver_interface=127.31.61.130
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:41:39.064229 14812 raft_consensus.cc:348] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:39.065129 14812 raft_consensus.cc:729] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f699173dce3457ba9d34619d8d932e8, State: Initialized, Role: FOLLOWER
I20260812 16:41:39.065760 14812 consensus_queue.cc:260] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:39.067589 14812 ts_tablet_manager.cc:1428] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260812 16:41:39.067816 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.067271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:35035)
0812 16:41:39.067455 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.067554 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 16:41:39.068821 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.068432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:38773)
0812 16:41:39.068631 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.068701 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260812 16:41:39.401234 14925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:39.401500 14925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:39.401603 14925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:39.408732 14925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:39.408839 14925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:39.408964 14925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:41:39.418467 14925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:43715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=34131
--tserver_master_addrs=127.31.61.190:36469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:39.419737 14925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:39.420989 14925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:39.433648 14930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:39.433718 14933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:39.433739 14931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:39.434525 14925 server_base.cc:1034] running on GCE node
I20260812 16:41:39.435199 14925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:39.436219 14925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:39.437435 14925 hybrid_clock.cc:648] HybridClock initialized: now 1786552899437377 us; error 49 us; skew 500 ppm
I20260812 16:41:39.437773 14925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:39 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:39.440953 14925 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:41:39.442453 14925 webserver.cc:460] Webserver started at http://127.31.61.130:34131/ using document root <none> and password file <none>
I20260812 16:41:39.443123 14925 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:39.443243 14925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:39.448607 14925 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:41:39.453701 14940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:39.455120 14925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:41:39.455294 14925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "f448b863e37646c7affecd3f8436fc79"
format_stamp: "Formatted at 2026-08-12 16:41:33 on dist-test-slave-f7th"
server_key: "d770db2d0bc8c17f716b4f4c476cba99"
server_key_iv: "5f5b04ae823c13e6f562ac363942dac7"
server_key_version: "encryptionkey@0"
I20260812 16:41:39.456354 14925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:39.486021 14925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:39.490659 14925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:39.490933 14925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:39.491974 14925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:39.493350 14925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:39.493450 14925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:39.493563 14925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:39.493631 14925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:39.524260 14925 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:43715
I20260812 16:41:39.524328 15052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:43715 every 8 connection(s)
I20260812 16:41:39.525713 14925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:41:39.530112 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 14925
I20260812 16:41:39.530244 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.528523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:35687)
0812 16:41:39.529195 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.529416 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":241,"threads_started":1}
I20260812 16:41:39.530799 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:34351
--local_ip_for_outbound_sockets=127.31.61.131
--tserver_master_addrs=127.31.61.190:36469
--webserver_port=44989
--webserver_interface=127.31.61.131
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:39.530978 15053 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:36469 (0 consecutive failures): Network error: Failed to ping master at 127.31.61.190:36469: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
I20260812 16:41:39.531827 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.531467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:38321)
0812 16:41:39.531654 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.531726 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 16:41:39.532626 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.532305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:48571)
0812 16:41:39.532444 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.532516 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 16:41:39.532850 15053 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:41:39.852825 15056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:39.853154 15056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:39.853283 15056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:39.860934 15056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:39.861059 15056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:39.861225 15056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:41:39.871901 15056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:34351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=44989
--tserver_master_addrs=127.31.61.190:36469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:39.873340 15056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:39.874663 15056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:39.887573 15061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:39.887626 15062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:39.887662 15064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:39.888476 15056 server_base.cc:1034] running on GCE node
I20260812 16:41:39.889128 15056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:39.890069 15056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:39.891265 15056 hybrid_clock.cc:648] HybridClock initialized: now 1786552899891208 us; error 48 us; skew 500 ppm
I20260812 16:41:39.891583 15056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:39 dist-test-slave-f7th krb5kdc[13590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:39.894500 15056 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:41:39.895924 15056 webserver.cc:460] Webserver started at http://127.31.61.131:44989/ using document root <none> and password file <none>
I20260812 16:41:39.896580 15056 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:39.896697 15056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:39.902034 15056 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 16:41:39.904798 15071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:39.906215 15056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:41:39.906387 15056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "50f50261d31d4cd186c1b69e905ba050"
format_stamp: "Formatted at 2026-08-12 16:41:34 on dist-test-slave-f7th"
server_key: "2518398b31673fc7b25fd8f875176e95"
server_key_iv: "ec709db24df070b232c8a50afaae4be3"
server_key_version: "encryptionkey@0"
I20260812 16:41:39.907375 15056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:39.926430 15056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:39.930826 15056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:39.931109 15056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:39.932201 15056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:39.933585 15056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:39.933686 15056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:39.933794 15056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:39.933861 15056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:39.966254 15056 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:34351
I20260812 16:41:39.966303 15185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:34351 every 8 connection(s)
I20260812 16:41:39.967765 15056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:41:39.972504 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.970660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:51635)
0812 16:41:39.971402 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.971670 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":282,"threads_started":1}
W20260812 16:41:39.973258 15186 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:36469 (0 consecutive failures): Network error: Failed to ping master at 127.31.61.190:36469: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
I20260812 16:41:39.974071 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.973740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:59775)
0812 16:41:39.973889 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.973960 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 16:41:39.974829 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 15056
I20260812 16:41:39.975023 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:39.974691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:37401)
0812 16:41:39.974821 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:39.974897 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 16:41:39.975242 15186 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 16:41:40.070470 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:40.069654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:46275)
0812 16:41:40.070145 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:40.070270 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":245,"threads_started":1}
I20260812 16:41:40.336083 15191 raft_consensus.cc:482] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:40.336354 15191 raft_consensus.cc:504] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:40.337936 15191 leader_election.cc:288] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715), 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351)
Aug 12 16:41:40 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
Aug 12 16:41:40 dist-test-slave-f7th krb5kdc[13590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:41:40.358980 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:40.338327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:34351 (local address 127.31.61.129:49549)
0812 16:41:40.340105 (+ 1778us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:40.340122 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:40.340257 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:40.342574 (+ 2317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:40.342585 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:40.343332 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:40.344075 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:40.344096 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:40.345696 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:40.345699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:40.347369 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:40.347385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:40.348377 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:40.348383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:40.348658 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:40.349359 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:40.349382 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:40.352252 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:40.355435 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:40.355445 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:40.355460 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:40.356045 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:40.356447 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:40.356452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:40.356456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:40.356671 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:40.357409 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:40.357417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:40.357774 (+ 357us) client_negotiation.cc:769] Sending connection context
0812 16:41:40.358053 (+ 279us) client_negotiation.cc:240] Negotiation successful
0812 16:41:40.358563 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1644,"mutex_wait_us":75,"spinlock_wait_cycles":30592,"thread_start_us":1466,"threads_started":1}
I20260812 16:41:40.359107 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:40.338789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:49549 (local address 127.31.61.131:34351)
0812 16:41:40.339839 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:40.339855 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:40.340255 (+ 400us) server_negotiation.cc:407] Connection header received
0812 16:41:40.341780 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:40.341789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:40.342128 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:40.342415 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:40.344262 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:40.345428 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:40.347504 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:40.348181 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:40.348326 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:40.352484 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:40.352491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:40.352503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:40.352543 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:40.355292 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:40.356175 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:40.356182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:40.356187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:40.356340 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:40.356778 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:40.356781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:40.356783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:40.357277 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:40.357452 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:40.358328 (+ 876us) server_negotiation.cc:299] Negotiation successful
0812 16:41:40.358685 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":672,"threads_started":1}
I20260812 16:41:40.359136 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:40.338123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:43715 (local address 127.31.61.129:52687)
0812 16:41:40.338320 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:40.338357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:40.338947 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:40.342575 (+ 3628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:40.342586 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:40.343317 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:40.344076 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:40.344096 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:40.345620 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:40.345626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:40.347373 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:40.347388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:40.348418 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:40.348433 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:40.348658 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:40.349404 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:40.349428 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:40.351917 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:40.355457 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:40.355462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:40.355467 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:40.356045 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:40.356465 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:40.356470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:40.356474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:40.356663 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:40.357409 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:40.357417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:40.357761 (+ 344us) client_negotiation.cc:769] Sending connection context
0812 16:41:40.358058 (+ 297us) client_negotiation.cc:240] Negotiation successful
0812 16:41:40.358705 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":380,"spinlock_wait_cycles":9088}
I20260812 16:41:40.360406 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:40.338434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:52687 (local address 127.31.61.130:43715)
0812 16:41:40.340288 (+ 1854us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:40.340302 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:40.340338 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:41:40.341708 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:40.341718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:40.342102 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:40.342380 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:40.344266 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:40.345423 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:40.347546 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:40.348208 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:40.348395 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:40.352234 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:40.352242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:40.352257 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:40.352305 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:40.355308 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:40.356175 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:40.356182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:40.356187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:40.356351 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:40.356774 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:40.356777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:40.356779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:40.357277 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:40.357454 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:40.359582 (+ 2128us) server_negotiation.cc:299] Negotiation successful
0812 16:41:40.359987 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1676,"thread_start_us":1568,"threads_started":1}
W20260812 16:41:40.362738 14806 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
W20260812 16:41:40.363237 14808 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
I20260812 16:41:40.363370 14808 leader_election.cc:302] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f699173dce3457ba9d34619d8d932e8; no voters: 50f50261d31d4cd186c1b69e905ba050, f448b863e37646c7affecd3f8436fc79
I20260812 16:41:40.363785 15191 raft_consensus.cc:2726] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:41:40.534284 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:40.533368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:59975)
0812 16:41:40.533889 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:40.534047 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":265,"threads_started":1}
I20260812 16:41:40.976568 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:40.975819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:57595)
0812 16:41:40.976243 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:40.976351 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":201,"threads_started":1}
I20260812 16:41:41.072187 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:41.071439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:58135)
0812 16:41:41.071941 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:41.072028 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":290,"threads_started":1}
I20260812 16:41:41.536230 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:41.535375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:39671)
0812 16:41:41.535956 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:41.536053 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":310,"threads_started":1}
I20260812 16:41:41.930920 15199 raft_consensus.cc:482] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:41.931134 15199 raft_consensus.cc:504] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:41.931849 15199 leader_election.cc:288] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715), 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351)
W20260812 16:41:41.932842 14806 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
W20260812 16:41:41.933007 14808 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
I20260812 16:41:41.933285 14808 leader_election.cc:302] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f699173dce3457ba9d34619d8d932e8; no voters: 50f50261d31d4cd186c1b69e905ba050, f448b863e37646c7affecd3f8436fc79
I20260812 16:41:41.933547 15199 raft_consensus.cc:2726] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:41:41.978204 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:41.977466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:42775)
0812 16:41:41.977917 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:41.978005 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":225,"threads_started":1}
I20260812 16:41:42.074261 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:42.073291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:45795)
0812 16:41:42.073925 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:42.074061 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":295,"threads_started":1}
I20260812 16:41:42.538100 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:42.537222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:45171)
0812 16:41:42.537743 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:42.537871 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":256,"threads_started":1}
I20260812 16:41:42.980086 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:42.979269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:38107)
0812 16:41:42.979785 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:42.979891 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":269,"threads_started":1}
I20260812 16:41:43.075943 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:43.075223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:50143)
0812 16:41:43.075687 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:43.075775 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":226,"threads_started":1}
I20260812 16:41:43.539835 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:43.539070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:57123)
0812 16:41:43.539536 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:43.539654 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":227,"threads_started":1}
I20260812 16:41:43.981827 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:43.981019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:44137)
0812 16:41:43.981537 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:43.981634 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":266,"threads_started":1}
I20260812 16:41:44.078156 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:44.077258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:47053)
0812 16:41:44.077878 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:44.077982 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":348,"threads_started":1}
I20260812 16:41:44.541891 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:44.540904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:50091)
0812 16:41:44.541558 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:44.541661 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":323,"threads_started":1}
I20260812 16:41:44.983522 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:44.982773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:60183)
0812 16:41:44.983242 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:44.983344 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":257,"threads_started":1}
I20260812 16:41:45.080185 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:45.079268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:42205)
0812 16:41:45.079868 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:45.079979 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":341,"threads_started":1}
I20260812 16:41:45.289769 15211 raft_consensus.cc:482] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:45.290012 15211 raft_consensus.cc:504] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:45.290750 15211 leader_election.cc:288] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715), 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351)
W20260812 16:41:45.291777 14806 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
W20260812 16:41:45.291960 14808 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
I20260812 16:41:45.292117 14808 leader_election.cc:302] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f699173dce3457ba9d34619d8d932e8; no voters: 50f50261d31d4cd186c1b69e905ba050, f448b863e37646c7affecd3f8436fc79
I20260812 16:41:45.292358 15211 raft_consensus.cc:2726] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:41:45.543723 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:45.542897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:39075)
0812 16:41:45.543404 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:45.543496 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":262,"threads_started":1}
I20260812 16:41:45.985268 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:45.984502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:35785)
0812 16:41:45.984992 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:45.985081 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":261,"threads_started":1}
I20260812 16:41:46.081876 15214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:46.081110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:55737)
0812 16:41:46.081585 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:46.081681 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260812 16:41:46.545481 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:46.544654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:36019)
0812 16:41:46.545206 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:46.545290 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":239,"threads_started":1}
I20260812 16:41:46.987181 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:46.986306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:60829)
0812 16:41:46.986881 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:46.986969 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":289,"threads_started":1}
I20260812 16:41:47.083644 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:47.082743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:33891)
0812 16:41:47.083329 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:47.083416 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":366,"threads_started":1}
I20260812 16:41:47.547175 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:47.546385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:59093)
0812 16:41:47.546867 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:47.546961 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260812 16:41:47.988940 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:47.988141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:47301)
0812 16:41:47.988661 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:47.988756 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":242,"threads_started":1}
I20260812 16:41:48.085387 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:48.084575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:54245)
0812 16:41:48.085101 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:48.085197 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":338,"threads_started":1}
I20260812 16:41:48.548826 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:48.548079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:43011)
0812 16:41:48.548576 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:48.548669 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":223,"threads_started":1}
I20260812 16:41:48.850620 15223 raft_consensus.cc:482] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:48.850835 15223 raft_consensus.cc:504] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:48.851536 15223 leader_election.cc:288] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715), 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351)
W20260812 16:41:48.852480 14806 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
W20260812 16:41:48.852669 14808 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
I20260812 16:41:48.852763 14808 leader_election.cc:302] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f699173dce3457ba9d34619d8d932e8; no voters: 50f50261d31d4cd186c1b69e905ba050, f448b863e37646c7affecd3f8436fc79
I20260812 16:41:48.852994 15223 raft_consensus.cc:2726] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:41:48.990541 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:48.989822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:46731)
0812 16:41:48.990272 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:48.990359 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":209,"threads_started":1}
I20260812 16:41:49.087361 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:49.086410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:45539)
0812 16:41:49.087022 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:49.087159 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":307,"threads_started":1}
I20260812 16:41:49.550516 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:49.549745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:50075)
0812 16:41:49.550221 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:49.550306 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":267,"threads_started":1}
I20260812 16:41:49.992416 15227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:49.991570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:46527)
0812 16:41:49.992092 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:49.992195 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":246,"threads_started":1}
I20260812 16:41:50.089154 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:50.088290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:58965)
0812 16:41:50.088854 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:50.088960 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":255,"threads_started":1}
I20260812 16:41:50.552309 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:50.551472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:46821)
0812 16:41:50.552028 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:50.552138 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
I20260812 16:41:50.994118 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:50.993375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:52365)
0812 16:41:50.993859 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:50.993952 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":235,"threads_started":1}
I20260812 16:41:51.090799 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:51.090054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:33069)
0812 16:41:51.090494 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:51.090609 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
I20260812 16:41:51.554136 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:51.553384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:35879)
0812 16:41:51.553848 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:51.553933 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":230,"threads_started":1}
I20260812 16:41:51.995987 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:51.995145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:56521)
0812 16:41:51.995711 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:51.995805 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":257,"threads_started":1}
I20260812 16:41:52.092623 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:52.091865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:55425)
0812 16:41:52.092332 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:52.092425 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260812 16:41:52.556118 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:52.555169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:48603)
0812 16:41:52.555777 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:52.555895 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":297,"threads_started":1}
I20260812 16:41:52.997953 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:52.997132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:48211)
0812 16:41:52.997674 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:52.997775 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":260,"threads_started":1}
I20260812 16:41:53.094435 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:53.093619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:52023)
0812 16:41:53.094154 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:53.094249 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":225,"threads_started":1}
I20260812 16:41:53.557888 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:53.557097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:40611)
0812 16:41:53.557582 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:53.557681 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
I20260812 16:41:53.999789 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:53.998917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:53565)
0812 16:41:53.999419 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:53.999534 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":246,"threads_started":1}
I20260812 16:41:54.096643 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:54.095817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.129:33517)
0812 16:41:54.096350 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:54.096440 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":274,"threads_started":1}
I20260812 16:41:54.560103 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:54.559211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.130:48621)
0812 16:41:54.559814 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:54.559921 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":366,"threads_started":1}
I20260812 16:41:54.840909 15242 raft_consensus.cc:482] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:54.841104 15242 raft_consensus.cc:504] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f699173dce3457ba9d34619d8d932e8" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 35119 } } peers { permanent_uuid: "f448b863e37646c7affecd3f8436fc79" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 43715 } } peers { permanent_uuid: "50f50261d31d4cd186c1b69e905ba050" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 34351 } }
I20260812 16:41:54.841689 15242 leader_election.cc:288] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715), 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351)
W20260812 16:41:54.842592 14806 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50f50261d31d4cd186c1b69e905ba050 (127.31.61.131:34351): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
W20260812 16:41:54.842813 14808 leader_election.cc:341] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f448b863e37646c7affecd3f8436fc79 (127.31.61.130:43715): Not found: Tablet not found: 8b9e6a766ddf45a8beb5a618ba638cf0
I20260812 16:41:54.842939 14808 leader_election.cc:302] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f699173dce3457ba9d34619d8d932e8; no voters: 50f50261d31d4cd186c1b69e905ba050, f448b863e37646c7affecd3f8436fc79
I20260812 16:41:54.843183 15242 raft_consensus.cc:2726] T 8b9e6a766ddf45a8beb5a618ba638cf0 P 4f699173dce3457ba9d34619d8d932e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:41:54.987210 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 14754
I20260812 16:41:55.001700 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:55.000904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36469 (local address 127.31.61.131:35031)
0812 16:41:55.001409 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:55.001530 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
I20260812 16:41:55.002879 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:54.988303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.31.61.129:35119)
0812 16:41:54.989016 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:54.989027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:54.989059 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:41:54.989135 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:54.989144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:54.989426 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:54.989591 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:54.991185 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:54.992262 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:54.993545 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:54.994101 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:54.994218 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:54.997076 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:54.997092 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:54.997108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:54.997147 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:55.000154 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:55.000859 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:55.000867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:55.000869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:55.001014 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:55.001391 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:55.001395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:55.001397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:55.001761 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:55.002055 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:55.002371 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 16:41:55.002599 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":341,"threads_started":1}
I20260812 16:41:55.004238 14916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:55.096185 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 14754
I20260812 16:41:55.112835 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 14925
I20260812 16:41:55.126453 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:55.113732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.31.61.130:43715)
0812 16:41:55.114234 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:55.114238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:55.114255 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:41:55.114311 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:55.114315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:55.114372 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:55.114459 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:55.115880 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:55.116673 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:55.117901 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:55.118342 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:55.118436 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:55.121068 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:55.121073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:55.121077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:55.121134 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:55.123785 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:55.124511 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:55.124516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:55.124519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:55.124676 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:55.125128 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:55.125133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:55.125135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:55.125419 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:55.125639 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:55.125913 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 16:41:55.126101 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":280,"threads_started":1}
I20260812 16:41:55.127307 15048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:55.210852 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 14925
I20260812 16:41:55.226430 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 15056
I20260812 16:41:55.240892 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:55.227253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.31.61.131:34351)
0812 16:41:55.227793 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:55.227797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:55.227815 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:41:55.227872 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:55.227876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:55.227946 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:55.228035 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:55.229503 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:55.230297 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:55.231882 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:55.232349 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:55.232455 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:55.235261 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:55.235267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:55.235271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:55.235306 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:55.238068 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:55.238806 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:55.238810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:55.238812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:55.238981 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:55.239414 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:55.239418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:55.239420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:55.239781 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:55.240011 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:55.240331 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 16:41:55.240536 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":278,"threads_started":1}
I20260812 16:41:55.241576 15181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:55.326725 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 15056
2026-08-12T16:41:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24203 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:41:55 dist-test-slave-f7th krb5kdc[15252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:41:55 dist-test-slave-f7th krb5kdc[15252](info): set up 2 sockets
Aug 12 16:41:55 dist-test-slave-f7th krb5kdc[15252](info): commencing operation
krb5kdc: starting...
W20260812 16:41:56.375874 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:41:56 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:41:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:41:56.621441 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43821
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:56.960441 15265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:56.960753 15265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:56.960880 15265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:56.968310 15265 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 16:41:56.968427 15265 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 16:41:56.968468 15265 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 16:41:56.968538 15265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:41:56.968621 15265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:56.968705 15265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:56.968767 15265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:41:56.968837 15265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:41:56.979051 15265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:56.980448 15265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:56.981886 15265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:56.993933 15273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:56.994009 15270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:56.993988 15271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:56.994369 15265 server_base.cc:1034] running on GCE node
I20260812 16:41:56.995404 15265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:56.996901 15265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:56.998131 15265 hybrid_clock.cc:648] HybridClock initialized: now 1786552916998071 us; error 38 us; skew 500 ppm
I20260812 16:41:56.998488 15265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:57 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552917, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:57.006107 15265 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:41:57.007992 15265 webserver.cc:460] Webserver started at http://127.31.61.190:36071/ using document root <none> and password file <none>
I20260812 16:41:57.008723 15265 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:57.008831 15265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:57.009150 15265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:57.012279 15265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "2c4b57d6d4944e209c4da660438ce36f"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "bd9387c364fcb497ba0f90c5ee4110bb"
server_key_iv: "608d72f22c746bced61fda430399998d"
server_key_version: "encryptionkey@0"
I20260812 16:41:57.013072 15265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "2c4b57d6d4944e209c4da660438ce36f"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "bd9387c364fcb497ba0f90c5ee4110bb"
server_key_iv: "608d72f22c746bced61fda430399998d"
server_key_version: "encryptionkey@0"
I20260812 16:41:57.018586 15265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:41:57.022339 15280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:57.023808 15265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:41:57.023970 15265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "2c4b57d6d4944e209c4da660438ce36f"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "bd9387c364fcb497ba0f90c5ee4110bb"
server_key_iv: "608d72f22c746bced61fda430399998d"
server_key_version: "encryptionkey@0"
I20260812 16:41:57.024191 15265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:57.056847 15265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:57.061231 15265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:57.061478 15265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:57.079764 15265 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:43821
I20260812 16:41:57.079787 15332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:43821 every 8 connection(s)
I20260812 16:41:57.081169 15265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:41:57.085016 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:57.085963 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 15265
I20260812 16:41:57.086166 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:41:57.086803 31990 external_mini_cluster.cc:1428] Setting key 97b9ade94ed69ebd9025baefc46b3a91
I20260812 16:41:57.099117 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Bootstrap starting.
Aug 12 16:41:57 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:57.102358 15333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:57.103768 15333 log.cc:824] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:57.107728 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: No bootstrap required, opened a new log
I20260812 16:41:57.112350 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:57.088797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.31.61.190:43821)
0812 16:41:57.089597 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:57.089612 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:57.089661 (+ 49us) server_negotiation.cc:407] Connection header received
0812 16:41:57.091445 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:57.091474 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:57.092011 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:57.092478 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:57.093925 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:57.095906 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:57.097365 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:57.098388 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:57.098572 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:57.102825 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:57.102844 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:57.102868 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:57.102915 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:57.107271 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:57.108232 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:57.108242 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:57.108251 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:57.108462 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:57.108919 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:57.108923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:57.108926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:57.109673 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:57.109987 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:57.110378 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 16:41:57.110859 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":329,"threads_started":1}
I20260812 16:41:57.116832 15333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:41:57.117206 15333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:57.117274 15333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4b57d6d4944e209c4da660438ce36f, State: Initialized, Role: FOLLOWER
I20260812 16:41:57.118129 15333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:41:57.118379 15333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:41:57.118521 15333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:41:57.118705 15333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:57.121039 15333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:41:57.121542 15333 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4b57d6d4944e209c4da660438ce36f; no voters:
I20260812 16:41:57.122112 15333 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:41:57.122216 15338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:41:57.123426 15338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 1 LEADER]: Becoming Leader. State: Replica: 2c4b57d6d4944e209c4da660438ce36f, State: Running, Role: LEADER
I20260812 16:41:57.123976 15338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:41:57.124291 15333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:41:57.126673 15339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c4b57d6d4944e209c4da660438ce36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } } }
I20260812 16:41:57.127076 15339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:57.127457 15340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4b57d6d4944e209c4da660438ce36f. Latest consensus state: current_term: 1 leader_uuid: "2c4b57d6d4944e209c4da660438ce36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } } }
I20260812 16:41:57.127763 15340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: This master's current role is: LEADER
I20260812 16:41:57.133957 15344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:41:57.139958 15344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:41:57.150614 15344 catalog_manager.cc:1347] Generated new cluster ID: 9f7fe40e2f8b4c3096849149814ec379
I20260812 16:41:57.150753 15344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:41:57.185230 15344 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:41:57.187422 15344 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:41:57.199391 15344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Generated new TSK 0
I20260812 16:41:57.200331 15344 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:41:57.283562 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43821
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:57.625686 15359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:57.626006 15359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:57.626288 15359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:57.634260 15359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:57.634383 15359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:57.634545 15359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:41:57.644816 15359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:57.646128 15359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:57.647447 15359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:57.660522 15364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:57.660558 15365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:57.660485 15367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:57.661767 15359 server_base.cc:1034] running on GCE node
I20260812 16:41:57.662367 15359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:57.663288 15359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:57.664506 15359 hybrid_clock.cc:648] HybridClock initialized: now 1786552917664456 us; error 35 us; skew 500 ppm
I20260812 16:41:57.664815 15359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:57 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552917, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:57.667896 15359 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:41:57.669332 15359 webserver.cc:460] Webserver started at http://127.31.61.129:41287/ using document root <none> and password file <none>
I20260812 16:41:57.669996 15359 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:57.670096 15359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:57.670398 15359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:57.673491 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "66f36981f7b64841b0af4a437c616b33"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "a55eaa57ba7deb15382d8dd895e9f69c"
server_key_iv: "78484392b1eee6dce3086535ff7d8b2c"
server_key_version: "encryptionkey@0"
I20260812 16:41:57.674292 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "66f36981f7b64841b0af4a437c616b33"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "a55eaa57ba7deb15382d8dd895e9f69c"
server_key_iv: "78484392b1eee6dce3086535ff7d8b2c"
server_key_version: "encryptionkey@0"
I20260812 16:41:57.679548 15359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 16:41:57.683244 15374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:57.684831 15359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:41:57.685011 15359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "66f36981f7b64841b0af4a437c616b33"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "a55eaa57ba7deb15382d8dd895e9f69c"
server_key_iv: "78484392b1eee6dce3086535ff7d8b2c"
server_key_version: "encryptionkey@0"
I20260812 16:41:57.685232 15359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:57.708432 15359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:57.712886 15359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:57.713184 15359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:57.714192 15359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:57.715535 15359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:57.715648 15359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 16:41:57.715756 15359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:57.715812 15359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:57.746822 15359 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:46077
I20260812 16:41:57.746872 15488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:46077 every 8 connection(s)
I20260812 16:41:57.748135 15359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:41:57.752154 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 15359
I20260812 16:41:57.752378 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:41:57.753064 31990 external_mini_cluster.cc:1428] Setting key 8f74807d9057c13f1207a7f2bfc3dcb6
Aug 12 16:41:57 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552917, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:57.769899 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:57.750696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:55435 (local address 127.31.61.190:43821)
0812 16:41:57.751192 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:57.751199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:57.752164 (+ 965us) server_negotiation.cc:407] Connection header received
0812 16:41:57.754416 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:57.754422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:57.754596 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:57.754699 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:57.756522 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:57.757594 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:57.759103 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:57.759668 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:57.759796 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:57.763354 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:57.763362 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:57.763366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:57.763407 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:57.766261 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:57.767210 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:57.767215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:57.767217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:57.767349 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:57.767900 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:57.767905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:57.767907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:57.768243 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:57.768395 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:57.769350 (+ 955us) server_negotiation.cc:299] Negotiation successful
0812 16:41:57.769622 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20260812 16:41:57.771242 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:57.751206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:55435)
0812 16:41:57.751995 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:57.752043 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:57.754121 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:57.754994 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:57.755006 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:57.755742 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:57.756336 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:57.756354 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:57.757737 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:57.757741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:57.758961 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:57.758973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:57.759869 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:57.759874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:57.760091 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:57.760751 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:57.760774 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:57.763101 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:57.766432 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:57.766443 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:57.766462 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:57.767078 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:57.767486 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:57.767491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:57.767494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:57.767776 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:57.768363 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:57.768371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:57.768827 (+ 456us) client_negotiation.cc:769] Sending connection context
0812 16:41:57.769173 (+ 346us) client_negotiation.cc:240] Negotiation successful
0812 16:41:57.769738 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":305,"threads_started":1}
I20260812 16:41:57.772945 15489 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43821
I20260812 16:41:57.773295 15489 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:57.774333 15489 heartbeater.cc:511] Master 127.31.61.190:43821 requested a full tablet report, sending...
I20260812 16:41:57.777153 15297 ts_manager.cc:194] Registered new tserver with Master: 66f36981f7b64841b0af4a437c616b33 (127.31.61.129:46077)
I20260812 16:41:57.778645 15297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:55435
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:41:57.815065 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43821
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:58.163304 15495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:58.163661 15495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:58.163805 15495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:58.171306 15495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:58.171454 15495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:58.171662 15495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:41:58.181900 15495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:58.183295 15495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:58.184693 15495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:58.198506 15503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:58.198627 15501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:58.198546 15500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:58.198945 15495 server_base.cc:1034] running on GCE node
I20260812 16:41:58.199743 15495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:58.200754 15495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:58.201982 15495 hybrid_clock.cc:648] HybridClock initialized: now 1786552918201930 us; error 45 us; skew 500 ppm
I20260812 16:41:58.202316 15495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:58 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552918, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:58.205607 15495 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:41:58.207219 15495 webserver.cc:460] Webserver started at http://127.31.61.130:34911/ using document root <none> and password file <none>
I20260812 16:41:58.208004 15495 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:58.208129 15495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:58.208424 15495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:58.211512 15495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "0ea790398f564aa8bb625575843b61c4"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "004817da9358c85d20fde203d351a4b8"
server_key_iv: "1379e857cb5de8f5b49564954d28090a"
server_key_version: "encryptionkey@0"
I20260812 16:41:58.212340 15495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "0ea790398f564aa8bb625575843b61c4"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "004817da9358c85d20fde203d351a4b8"
server_key_iv: "1379e857cb5de8f5b49564954d28090a"
server_key_version: "encryptionkey@0"
I20260812 16:41:58.217711 15495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 16:41:58.221495 15510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:58.223062 15495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:41:58.223244 15495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "0ea790398f564aa8bb625575843b61c4"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "004817da9358c85d20fde203d351a4b8"
server_key_iv: "1379e857cb5de8f5b49564954d28090a"
server_key_version: "encryptionkey@0"
I20260812 16:41:58.223453 15495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:58.245451 15495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:58.249861 15495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:58.250136 15495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:58.251186 15495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:58.252638 15495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:58.252719 15495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:58.252844 15495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:58.252898 15495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:58.285640 15495 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:33607
I20260812 16:41:58.285678 15622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:33607 every 8 connection(s)
I20260812 16:41:58.286969 15495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:41:58.291764 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 15495
I20260812 16:41:58.291953 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:41:58.292366 31990 external_mini_cluster.cc:1428] Setting key 2a623df0b972e2770ad7c829f97b8e92
Aug 12 16:41:58 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552918, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:58.311558 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.289691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:49271 (local address 127.31.61.190:43821)
0812 16:41:58.290198 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:58.290205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:58.291171 (+ 966us) server_negotiation.cc:407] Connection header received
0812 16:41:58.293181 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:58.293189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:58.293280 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:58.293402 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:58.295728 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.296735 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.298322 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.298898 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.299019 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.304058 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:58.304066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:58.304072 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:58.304126 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:58.308039 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:58.309142 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:58.309149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:58.309153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:58.309294 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:58.309762 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:58.309768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:58.309771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:58.310055 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:58.310203 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:58.310939 (+ 736us) server_negotiation.cc:299] Negotiation successful
0812 16:41:58.311178 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":225,"threads_started":1}
I20260812 16:41:58.312628 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.290234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:49271)
0812 16:41:58.290980 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:58.291019 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:58.292830 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:58.294326 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:58.294338 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:58.294903 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:58.295502 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.295521 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.296897 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.296901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.298164 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.298178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.299125 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.299130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.299335 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.300016 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:58.300045 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:58.302488 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:58.308273 (+ 5785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:58.308285 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:58.308306 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:58.308985 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:58.309413 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:58.309419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:58.309423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:58.309634 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:58.310182 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:58.310191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:58.310568 (+ 377us) client_negotiation.cc:769] Sending connection context
0812 16:41:58.310820 (+ 252us) client_negotiation.cc:240] Negotiation successful
0812 16:41:58.311393 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":283,"threads_started":1}
I20260812 16:41:58.314105 15623 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43821
I20260812 16:41:58.314437 15623 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:58.315354 15623 heartbeater.cc:511] Master 127.31.61.190:43821 requested a full tablet report, sending...
I20260812 16:41:58.317477 15297 ts_manager.cc:194] Registered new tserver with Master: 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
I20260812 16:41:58.318336 15297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:49271
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:41:58.356509 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43821
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:41:58.687549 15629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:41:58.687902 15629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:41:58.688052 15629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:41:58.695583 15629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:41:58.695766 15629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:41:58.695925 15629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:41:58.705853 15629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:41:58.707175 15629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:41:58.708552 15629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:41:58.723022 15634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:58.722983 15635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:41:58.723068 15637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:41:58.723649 15629 server_base.cc:1034] running on GCE node
I20260812 16:41:58.724337 15629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:41:58.725340 15629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:41:58.726555 15629 hybrid_clock.cc:648] HybridClock initialized: now 1786552918726495 us; error 49 us; skew 500 ppm
I20260812 16:41:58.726895 15629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:41:58 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552918, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:41:58.730283 15629 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:41:58.731925 15629 webserver.cc:460] Webserver started at http://127.31.61.131:43031/ using document root <none> and password file <none>
I20260812 16:41:58.732676 15629 fs_manager.cc:359] Metadata directory not provided
I20260812 16:41:58.732784 15629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:41:58.733088 15629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:41:58.736435 15629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "bc022496702b4488b578fa142036a618"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "35aa76a3432f66f82a50948f02018aec"
server_key_iv: "cdae57920267795bae92124a16f51508"
server_key_version: "encryptionkey@0"
I20260812 16:41:58.737226 15629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "bc022496702b4488b578fa142036a618"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "35aa76a3432f66f82a50948f02018aec"
server_key_iv: "cdae57920267795bae92124a16f51508"
server_key_version: "encryptionkey@0"
I20260812 16:41:58.743064 15629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:41:58.746809 15644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:58.748334 15629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:41:58.748493 15629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "bc022496702b4488b578fa142036a618"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "35aa76a3432f66f82a50948f02018aec"
server_key_iv: "cdae57920267795bae92124a16f51508"
server_key_version: "encryptionkey@0"
I20260812 16:41:58.748692 15629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:41:58.772701 15629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:41:58.778298 15629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:41:58.778637 15629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:41:58.779963 15629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:41:58.781433 15629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:41:58.781518 15629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:58.781611 15629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:41:58.781653 15629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:41:58.782027 15489 heartbeater.cc:503] Master 127.31.61.190:43821 was elected leader, sending a full tablet report...
I20260812 16:41:58.814136 15629 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:41667
I20260812 16:41:58.814203 15760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:41667 every 8 connection(s)
I20260812 16:41:58.815402 15629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:41:58.822361 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 15629
I20260812 16:41:58.822539 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:41:58.822992 31990 external_mini_cluster.cc:1428] Setting key 1f805c8969054cd2007abea5282ba0c6
Aug 12 16:41:58 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552918, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:41:58.835980 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.817960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:56005 (local address 127.31.61.190:43821)
0812 16:41:58.818450 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:58.818456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:58.819306 (+ 850us) server_negotiation.cc:407] Connection header received
0812 16:41:58.821339 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:58.821344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:58.821429 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:58.821535 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:58.823186 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.824154 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.825877 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.826394 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.826493 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.830146 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:58.830151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:58.830155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:58.830193 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:58.832783 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:58.833519 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:58.833523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:58.833525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:58.833694 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:58.834112 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:58.834117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:58.834120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:58.834426 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:58.834596 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:58.835335 (+ 739us) server_negotiation.cc:299] Negotiation successful
0812 16:41:58.835569 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":238,"threads_started":1}
I20260812 16:41:58.837095 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.818369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:56005)
0812 16:41:58.819132 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:58.819177 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:58.821021 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:58.821786 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:58.821796 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:58.822344 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:41:58.822974 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.823001 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.824298 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.824304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.825744 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.825761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.826593 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.826598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.826819 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.827508 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:41:58.827538 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:41:58.829905 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:41:58.832921 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:58.832932 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:58.832951 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:58.833396 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:58.833801 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:41:58.833806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:41:58.833810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:41:58.834000 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:41:58.834553 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:41:58.834562 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:41:58.834956 (+ 394us) client_negotiation.cc:769] Sending connection context
0812 16:41:58.835218 (+ 262us) client_negotiation.cc:240] Negotiation successful
0812 16:41:58.835780 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":287,"threads_started":1}
I20260812 16:41:58.838644 15761 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43821
I20260812 16:41:58.838972 15761 heartbeater.cc:464] Registering TS with master...
I20260812 16:41:58.839717 15761 heartbeater.cc:511] Master 127.31.61.190:43821 requested a full tablet report, sending...
I20260812 16:41:58.841166 15297 ts_manager.cc:194] Registered new tserver with Master: bc022496702b4488b578fa142036a618 (127.31.61.131:41667)
I20260812 16:41:58.842098 15297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:56005
I20260812 16:41:58.851300 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:41:58.871160 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.856140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.31.61.190:43821)
0812 16:41:58.856318 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:58.856323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:58.856734 (+ 411us) server_negotiation.cc:407] Connection header received
0812 16:41:58.856947 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:58.856952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:58.857019 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:58.857106 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:58.858694 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.859569 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.861062 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.861522 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.861629 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.864528 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:58.864534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:58.864537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:58.864573 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:58.867262 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:58.868523 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:58.868528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:58.868530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:58.868675 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:58.869477 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:58.869481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:58.869483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:58.869842 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:58.870017 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:58.870510 (+ 493us) server_negotiation.cc:299] Negotiation successful
0812 16:41:58.870907 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:41:58.876006 15297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:41:58.883059 15297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:41:58.913996 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.896159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:46077 (local address 127.0.0.1:56974)
0812 16:41:58.904539 (+ 8380us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:58.904555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:58.904714 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:58.905394 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:58.905402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:58.905434 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:58.905858 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.905875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.908073 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.908080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.910523 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.910544 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.912795 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.912802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.913035 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.913095 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 16:41:58.913238 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 16:41:58.913469 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8224,"mutex_wait_us":1,"thread_start_us":6348,"threads_started":1}
I20260812 16:41:58.914520 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.896750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.31.61.129:46077)
0812 16:41:58.904129 (+ 7379us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:58.904137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:58.904676 (+ 539us) server_negotiation.cc:407] Connection header received
0812 16:41:58.904928 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:58.904937 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:58.905165 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:58.905357 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:58.906081 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.907872 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.910717 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.912255 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.912391 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.912644 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:58.913666 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0812 16:41:58.913847 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7219,"thread_start_us":7046,"threads_started":1}
I20260812 16:41:58.918296 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.895899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:41667 (local address 127.0.0.1:56052)
0812 16:41:58.903042 (+ 7143us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:58.903105 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:58.903322 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:58.911032 (+ 7710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:58.911037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:58.911061 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:58.911392 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.911400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.913061 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.913065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.915266 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.915281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.917155 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.917163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.917382 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.917473 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 16:41:58.917558 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 16:41:58.917735 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6794,"mutex_wait_us":69,"thread_start_us":8340,"threads_started":1}
I20260812 16:41:58.918318 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.905589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:33607 (local address 127.0.0.1:60532)
0812 16:41:58.909163 (+ 3574us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:58.909188 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:58.909340 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:58.910204 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:58.910209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:58.910239 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:58.910633 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.910646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.912661 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.912666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.915012 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:58.915024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.917190 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.917198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:58.917398 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.917479 (+ 81us) client_negotiation.cc:769] Sending connection context
0812 16:41:58.917559 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 16:41:58.917735 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3405,"mutex_wait_us":48,"thread_start_us":3202,"threads_started":1}
I20260812 16:41:58.918799 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.905137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.31.61.130:33607)
0812 16:41:58.908694 (+ 3557us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:58.908704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:58.909490 (+ 786us) server_negotiation.cc:407] Connection header received
0812 16:41:58.909632 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:58.909642 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:58.909953 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:58.910850 (+ 897us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:58.911060 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.912489 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.915165 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.916652 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.916792 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.916993 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:58.917996 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0812 16:41:58.918182 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3398,"thread_start_us":3282,"threads_started":1}
I20260812 16:41:58.918762 15423 tablet_service.cc:1467] Processing CreateTablet for tablet f77d3264e48b4ab59b62e17ce9620741 (DEFAULT_TABLE table=test-table [id=e305a82c279a4ceda6230728bc28e372]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:58.919526 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:58.896097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.31.61.131:41667)
0812 16:41:58.910540 (+ 14443us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:58.910548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:58.910582 (+ 34us) server_negotiation.cc:407] Connection header received
0812 16:41:58.910673 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:58.910680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:58.910855 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:58.911020 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:58.911554 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.912883 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.915428 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:58.916886 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:58.918655 (+ 1769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:58.918899 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:58.918975 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 16:41:58.919151 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14267,"thread_start_us":14111,"threads_started":1}
I20260812 16:41:58.920218 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77d3264e48b4ab59b62e17ce9620741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:58.922259 15558 tablet_service.cc:1467] Processing CreateTablet for tablet f77d3264e48b4ab59b62e17ce9620741 (DEFAULT_TABLE table=test-table [id=e305a82c279a4ceda6230728bc28e372]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:58.923453 15558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77d3264e48b4ab59b62e17ce9620741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:58.927441 15692 tablet_service.cc:1467] Processing CreateTablet for tablet f77d3264e48b4ab59b62e17ce9620741 (DEFAULT_TABLE table=test-table [id=e305a82c279a4ceda6230728bc28e372]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:41:58.929391 15692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77d3264e48b4ab59b62e17ce9620741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:41:58.937597 15782 tablet_bootstrap.cc:492] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: Bootstrap starting.
I20260812 16:41:58.939589 15781 tablet_bootstrap.cc:492] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: Bootstrap starting.
I20260812 16:41:58.942514 15782 tablet_bootstrap.cc:654] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:58.943215 15781 tablet_bootstrap.cc:654] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:58.943765 15782 log.cc:824] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:58.944496 15781 log.cc:824] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:58.946475 15782 tablet_bootstrap.cc:492] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: No bootstrap required, opened a new log
I20260812 16:41:58.946863 15782 ts_tablet_manager.cc:1397] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260812 16:41:58.950321 15786 tablet_bootstrap.cc:492] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: Bootstrap starting.
I20260812 16:41:58.950588 15781 tablet_bootstrap.cc:492] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: No bootstrap required, opened a new log
I20260812 16:41:58.950907 15781 ts_tablet_manager.cc:1397] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260812 16:41:58.954264 15786 tablet_bootstrap.cc:654] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: Neither blocks nor log segments found. Creating new log.
I20260812 16:41:58.955708 15786 log.cc:824] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: Log is configured to *not* fsync() on all Append() calls
I20260812 16:41:58.956911 15782 raft_consensus.cc:348] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:41:58.957324 15782 raft_consensus.cc:374] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:58.957441 15782 raft_consensus.cc:729] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f36981f7b64841b0af4a437c616b33, State: Initialized, Role: FOLLOWER
I20260812 16:41:58.958130 15782 consensus_queue.cc:260] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:41:58.959023 15781 raft_consensus.cc:348] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:41:58.959357 15786 tablet_bootstrap.cc:492] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: No bootstrap required, opened a new log
I20260812 16:41:58.959373 15781 raft_consensus.cc:374] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:58.959467 15781 raft_consensus.cc:729] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea790398f564aa8bb625575843b61c4, State: Initialized, Role: FOLLOWER
I20260812 16:41:58.959723 15786 ts_tablet_manager.cc:1397] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260812 16:41:58.960038 15781 consensus_queue.cc:260] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:41:58.962850 15623 heartbeater.cc:503] Master 127.31.61.190:43821 was elected leader, sending a full tablet report...
I20260812 16:41:58.963322 15781 ts_tablet_manager.cc:1428] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260812 16:41:58.966130 15782 ts_tablet_manager.cc:1428] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
I20260812 16:41:58.971676 15786 raft_consensus.cc:348] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:41:58.972082 15786 raft_consensus.cc:374] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:41:58.972157 15786 raft_consensus.cc:729] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc022496702b4488b578fa142036a618, State: Initialized, Role: FOLLOWER
I20260812 16:41:58.972674 15786 consensus_queue.cc:260] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:41:58.973790 15761 heartbeater.cc:503] Master 127.31.61.190:43821 was elected leader, sending a full tablet report...
I20260812 16:41:58.974431 15786 ts_tablet_manager.cc:1428] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: Time spent starting tablet: real 0.015s user 0.006s sys 0.006s
W20260812 16:41:59.000703 15490 tablet.cc:2369] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:41:59.039119 15624 tablet.cc:2369] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:41:59.066975 15762 tablet.cc:2369] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:41:59.260437 15788 raft_consensus.cc:482] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:41:59.260761 15788 raft_consensus.cc:504] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:41:59.262670 15788 leader_election.cc:288] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc022496702b4488b578fa142036a618 (127.31.61.131:41667), 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
I20260812 16:41:59.269856 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:59.263119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:60583 (local address 127.31.61.131:41667)
0812 16:41:59.263311 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:59.263318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:59.263930 (+ 612us) server_negotiation.cc:407] Connection header received
0812 16:41:59.264161 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:59.264167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:59.264249 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:59.264339 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:59.264929 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.265889 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.267894 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.268924 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.269047 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:59.269101 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:59.269484 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 16:41:59.269601 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:41:59.269914 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:59.263122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:41667 (local address 127.31.61.129:60583)
0812 16:41:59.263813 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:59.263829 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:59.263994 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:59.264409 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:59.264413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:59.264442 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:59.264784 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:59.264798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.266002 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.266008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:59.267691 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:59.267704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.269121 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.269128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:59.269299 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:59.269336 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:41:59.269435 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 16:41:59.269577 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":379,"threads_started":1}
I20260812 16:41:59.270498 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:59.264114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:48393 (local address 127.31.61.130:33607)
0812 16:41:59.264315 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:59.264320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:59.264713 (+ 393us) server_negotiation.cc:407] Connection header received
0812 16:41:59.264846 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:59.264851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:59.264926 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:59.264989 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:41:59.265620 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.266576 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.268563 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.269518 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.269658 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:59.269726 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:59.270127 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 16:41:59.270248 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:41:59.271024 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:59.263924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:33607 (local address 127.31.61.129:48393)
0812 16:41:59.264603 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0812 16:41:59.264620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:41:59.264732 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:41:59.265030 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:41:59.265034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:41:59.265058 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:41:59.265416 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:59.265427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.266733 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.266738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:59.268340 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:41:59.268355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.269760 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.269765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:41:59.269930 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:59.269960 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 16:41:59.270694 (+ 734us) client_negotiation.cc:240] Negotiation successful
0812 16:41:59.270800 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":300,"threads_started":1}
I20260812 16:41:59.271077 15712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77d3264e48b4ab59b62e17ce9620741" candidate_uuid: "66f36981f7b64841b0af4a437c616b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc022496702b4488b578fa142036a618" is_pre_election: true
I20260812 16:41:59.271634 15712 raft_consensus.cc:2454] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f36981f7b64841b0af4a437c616b33 in term 0.
I20260812 16:41:59.271886 15578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77d3264e48b4ab59b62e17ce9620741" candidate_uuid: "66f36981f7b64841b0af4a437c616b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea790398f564aa8bb625575843b61c4" is_pre_election: true
I20260812 16:41:59.272269 15376 leader_election.cc:302] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f36981f7b64841b0af4a437c616b33, bc022496702b4488b578fa142036a618; no voters:
I20260812 16:41:59.272396 15578 raft_consensus.cc:2454] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f36981f7b64841b0af4a437c616b33 in term 0.
I20260812 16:41:59.272665 15788 raft_consensus.cc:2781] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:41:59.272801 15788 raft_consensus.cc:482] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:41:59.272902 15788 raft_consensus.cc:3037] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:59.275072 15788 raft_consensus.cc:504] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:41:59.275736 15788 leader_election.cc:288] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 1 election: Requested vote from peers bc022496702b4488b578fa142036a618 (127.31.61.131:41667), 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
I20260812 16:41:59.276764 15712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77d3264e48b4ab59b62e17ce9620741" candidate_uuid: "66f36981f7b64841b0af4a437c616b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc022496702b4488b578fa142036a618"
I20260812 16:41:59.276784 15578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77d3264e48b4ab59b62e17ce9620741" candidate_uuid: "66f36981f7b64841b0af4a437c616b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea790398f564aa8bb625575843b61c4"
I20260812 16:41:59.276980 15712 raft_consensus.cc:3037] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:59.277156 15578 raft_consensus.cc:3037] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:41:59.279775 15712 raft_consensus.cc:2454] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f36981f7b64841b0af4a437c616b33 in term 1.
I20260812 16:41:59.279838 15578 raft_consensus.cc:2454] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f36981f7b64841b0af4a437c616b33 in term 1.
I20260812 16:41:59.280261 15376 leader_election.cc:302] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f36981f7b64841b0af4a437c616b33, bc022496702b4488b578fa142036a618; no voters:
I20260812 16:41:59.280567 15788 raft_consensus.cc:2781] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:41:59.281208 15788 raft_consensus.cc:686] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 1 LEADER]: Becoming Leader. State: Replica: 66f36981f7b64841b0af4a437c616b33, State: Running, Role: LEADER
I20260812 16:41:59.281965 15788 consensus_queue.cc:237] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:41:59.286618 15297 catalog_manager.cc:5495] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f36981f7b64841b0af4a437c616b33 (127.31.61.129). New cstate: current_term: 1 leader_uuid: "66f36981f7b64841b0af4a437c616b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } health_report { overall_health: UNKNOWN } } }
I20260812 16:41:59.380883 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:59.375198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.31.61.129:46077)
0812 16:41:59.375423 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:59.375431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:59.375549 (+ 118us) server_negotiation.cc:407] Connection header received
0812 16:41:59.375812 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:59.375816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:59.375898 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:59.376058 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:41:59.376467 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.377198 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.378745 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.379156 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.379264 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:59.379770 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:41:59.380288 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:41:59.380420 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:59.380528 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 16:41:59.380613 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 16:41:59.389086 15712 raft_consensus.cc:1261] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [term 1 FOLLOWER]: Refusing update from remote peer 66f36981f7b64841b0af4a437c616b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:59.389082 15578 raft_consensus.cc:1261] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [term 1 FOLLOWER]: Refusing update from remote peer 66f36981f7b64841b0af4a437c616b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:41:59.390200 15793 consensus_queue.cc:1035] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:41:59.390611 15788 consensus_queue.cc:1035] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:41:59.408551 15800 mvcc.cc:205] Tried to move back new op lower bound from 7317720757807714304 to 7317720757383036928. Current Snapshot: MvccSnapshot[applied={T|T < 7317720757807714304}]
I20260812 16:41:59.417393 15296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:41:59.417722 15296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:41:59.421665 15296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Sending DeleteTablet for 3 replicas of tablet f77d3264e48b4ab59b62e17ce9620741
I20260812 16:41:59.422931 15423 tablet_service.cc:1514] Processing DeleteTablet for tablet f77d3264e48b4ab59b62e17ce9620741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:59 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:56974
I20260812 16:41:59.422960 15558 tablet_service.cc:1514] Processing DeleteTablet for tablet f77d3264e48b4ab59b62e17ce9620741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:59 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:60532
I20260812 16:41:59.423223 15692 tablet_service.cc:1514] Processing DeleteTablet for tablet f77d3264e48b4ab59b62e17ce9620741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:41:59 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:56052
I20260812 16:41:59.423399 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 15359
I20260812 16:41:59.438844 15810 tablet_replica.cc:330] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: stopping tablet replica
I20260812 16:41:59.438887 15808 tablet_replica.cc:330] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: stopping tablet replica
I20260812 16:41:59.439160 15809 tablet_replica.cc:330] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: stopping tablet replica
I20260812 16:41:59.439404 15810 raft_consensus.cc:2229] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:41:59.439591 15809 raft_consensus.cc:2229] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:41:59.439683 15808 raft_consensus.cc:2229] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:41:59.440014 15810 raft_consensus.cc:2258] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:59.440117 15809 raft_consensus.cc:2258] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:59.440582 15808 raft_consensus.cc:2258] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:41:59.442548 15810 ts_tablet_manager.cc:1905] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:59.442795 15808 ts_tablet_manager.cc:1905] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 16:41:59 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:41:59.450392 15810 ts_tablet_manager.cc:1918] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:59.450584 15810 log.cc:1196] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/f77d3264e48b4ab59b62e17ce9620741
I20260812 16:41:59.450825 15808 ts_tablet_manager.cc:1918] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:59.451256 15808 log.cc:1196] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/f77d3264e48b4ab59b62e17ce9620741
I20260812 16:41:59.451267 15810 ts_tablet_manager.cc:1939] T f77d3264e48b4ab59b62e17ce9620741 P bc022496702b4488b578fa142036a618: Deleting consensus metadata
I20260812 16:41:59.451927 15808 ts_tablet_manager.cc:1939] T f77d3264e48b4ab59b62e17ce9620741 P 66f36981f7b64841b0af4a437c616b33: Deleting consensus metadata
I20260812 16:41:59.453126 15809 ts_tablet_manager.cc:1905] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:41:59.453382 15282 catalog_manager.cc:4841] TS bc022496702b4488b578fa142036a618 (127.31.61.131:41667): tablet f77d3264e48b4ab59b62e17ce9620741 (table test-table [id=e305a82c279a4ceda6230728bc28e372]) successfully deleted
I20260812 16:41:59.454526 15283 catalog_manager.cc:4841] TS 66f36981f7b64841b0af4a437c616b33 (127.31.61.129:46077): tablet f77d3264e48b4ab59b62e17ce9620741 (table test-table [id=e305a82c279a4ceda6230728bc28e372]) successfully deleted
I20260812 16:41:59.458575 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:59.424694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.31.61.129:46077)
0812 16:41:59.424878 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:59.424885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:59.435242 (+ 10357us) server_negotiation.cc:407] Connection header received
0812 16:41:59.435562 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:59.435569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:59.435688 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:59.435770 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:59.440559 (+ 4789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.441788 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.443799 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.444530 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.444633 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:59.450480 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:59.450496 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:59.450517 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:59.450598 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:59.455366 (+ 4768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:59.456237 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:59.456244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:59.456247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:59.456409 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:59.456917 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:59.456921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:59.456923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:59.457271 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:59.457571 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:59.457945 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 16:41:59.458186 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 16:41:59.459227 15483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:59.460915 15809 ts_tablet_manager.cc:1918] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:41:59.461126 15809 log.cc:1196] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/f77d3264e48b4ab59b62e17ce9620741
I20260812 16:41:59.461671 15809 ts_tablet_manager.cc:1939] T f77d3264e48b4ab59b62e17ce9620741 P 0ea790398f564aa8bb625575843b61c4: Deleting consensus metadata
I20260812 16:41:59.463810 15282 catalog_manager.cc:4841] TS 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607): tablet f77d3264e48b4ab59b62e17ce9620741 (table test-table [id=e305a82c279a4ceda6230728bc28e372]) successfully deleted
I20260812 16:41:59.564494 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 15359
I20260812 16:41:59.582647 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 15495
Aug 12 16:41:59 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:41:59.598099 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:59.583375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.31.61.130:33607)
0812 16:41:59.583556 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:59.583562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:59.583581 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:41:59.583848 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:59.583852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:59.583918 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:59.584001 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:59.585367 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.586098 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.587454 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.587904 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.588016 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:59.592200 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:59.592214 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:59.592230 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:59.592266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:59.595158 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:59.595965 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:59.595973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:59.595975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:59.596123 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:59.596496 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:59.596500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:59.596502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:59.596894 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:59.597222 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:59.597508 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 16:41:59.597730 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:41:59.598762 15618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:59.689431 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 15495
I20260812 16:41:59.704039 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 15629
Aug 12 16:41:59 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:41:59.718904 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:41:59.704846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.31.61.131:41667)
0812 16:41:59.705005 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 16:41:59.705009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:41:59.705048 (+ 39us) server_negotiation.cc:407] Connection header received
0812 16:41:59.705101 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:41:59.705105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:41:59.705166 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:41:59.705238 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:41:59.706528 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.707252 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.708518 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:41:59.708893 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:41:59.708991 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:41:59.713214 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:41:59.713231 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:41:59.713249 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:41:59.713299 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:41:59.716049 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:59.716738 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:59.716746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:59.716748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:59.716897 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:41:59.717250 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:41:59.717255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:41:59.717257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:41:59.717637 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:41:59.717879 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 16:41:59.718330 (+ 451us) server_negotiation.cc:299] Negotiation successful
0812 16:41:59.718524 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 16:41:59.719519 15755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:59.807900 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 15629
I20260812 16:41:59.822563 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 15265
I20260812 16:41:59.823733 15328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:41:59.933681 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 15265
I20260812 16:42:19.953001 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43821
--webserver_interface=127.31.61.190
--webserver_port=36071
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:42:20.283658 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:42:20.283952 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:42:20.284065 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:42:20.291487 15816 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 16:42:20.291616 15816 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 16:42:20.291657 15816 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 16:42:20.291711 15816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:42:20.291783 15816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:42:20.291877 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:42:20.291927 15816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:42:20.291975 15816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:42:20.302172 15816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=36071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:42:20.303478 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:42:20.304872 15816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:42:20.330093 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:20.330134 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:20.330157 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:42:20.330675 15816 server_base.cc:1034] running on GCE node
I20260812 16:42:20.331728 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:42:20.333633 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:42:20.335035 15816 hybrid_clock.cc:648] HybridClock initialized: now 1786552940334941 us; error 57 us; skew 500 ppm
I20260812 16:42:20.335642 15816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:42:20 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552940, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:42:20.345551 15816 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:42:20.347321 15816 webserver.cc:460] Webserver started at http://127.31.61.190:36071/ using document root <none> and password file <none>
I20260812 16:42:20.348177 15816 fs_manager.cc:359] Metadata directory not provided
I20260812 16:42:20.348295 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:42:20.355037 15816 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 16:42:20.358398 15833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:20.359908 15816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:42:20.360082 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "2c4b57d6d4944e209c4da660438ce36f"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "bd9387c364fcb497ba0f90c5ee4110bb"
server_key_iv: "608d72f22c746bced61fda430399998d"
server_key_version: "encryptionkey@0"
I20260812 16:42:20.361166 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:42:20.377660 15816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:42:20.382145 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:42:20.382438 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:42:20.401765 15816 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:43821
I20260812 16:42:20.401795 15888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:43821 every 8 connection(s)
I20260812 16:42:20.403352 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:42:20.408325 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 15816
I20260812 16:42:20.409356 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:46077
--local_ip_for_outbound_sockets=127.31.61.129
--tserver_master_addrs=127.31.61.190:43821
--webserver_port=41287
--webserver_interface=127.31.61.129
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:42:20.411628 15889 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:42:20.416214 15889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Bootstrap starting.
I20260812 16:42:20.437580 15889 log.cc:824] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:42:20.451217 15889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:42:20.451839 15889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Bootstrap complete.
I20260812 16:42:20.461799 15889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:42:20.462725 15889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4b57d6d4944e209c4da660438ce36f, State: Initialized, Role: FOLLOWER
I20260812 16:42:20.463416 15889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:42:20.463683 15889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:42:20.463810 15889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:42:20.463981 15889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 1 FOLLOWER]: Advancing to term 2
I20260812 16:42:20.468309 15889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:42:20.468864 15889 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4b57d6d4944e209c4da660438ce36f; no voters:
I20260812 16:42:20.469513 15889 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 16:42:20.469668 15893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 2 FOLLOWER]: Leader election won for term 2
I20260812 16:42:20.472245 15889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:42:20.475898 15893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 2 LEADER]: Becoming Leader. State: Replica: 2c4b57d6d4944e209c4da660438ce36f, State: Running, Role: LEADER
I20260812 16:42:20.476640 15893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:42:20.479806 15894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c4b57d6d4944e209c4da660438ce36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } } }
I20260812 16:42:20.479830 15895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4b57d6d4944e209c4da660438ce36f. Latest consensus state: current_term: 2 leader_uuid: "2c4b57d6d4944e209c4da660438ce36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } } }
I20260812 16:42:20.480271 15894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: This master's current role is: LEADER
I20260812 16:42:20.480276 15895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: This master's current role is: LEADER
I20260812 16:42:20.492100 15905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:42:20.500604 15905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:42:20.502192 15905 catalog_manager.cc:1259] Loaded cluster ID: 9f7fe40e2f8b4c3096849149814ec379
I20260812 16:42:20.502316 15905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:42:20.506589 15905 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:42:20.508194 15905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Loaded TSK: 0
I20260812 16:42:20.546711 15905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Generated new TSK 1
I20260812 16:42:20.547152 15905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:42:20.779735 15891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:42:20.780149 15891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:42:20.780310 15891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:42:20.789839 15891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:42:20.789978 15891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:42:20.790148 15891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:42:20.800981 15891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:46077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=41287
--tserver_master_addrs=127.31.61.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:42:20.802398 15891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:42:20.803838 15891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:42:20.819473 15916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:20.819620 15917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:20.820288 15919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:42:20.820763 15891 server_base.cc:1034] running on GCE node
I20260812 16:42:20.821472 15891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:42:20.822535 15891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:42:20.823732 15891 hybrid_clock.cc:648] HybridClock initialized: now 1786552940823652 us; error 67 us; skew 500 ppm
I20260812 16:42:20.824085 15891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:42:20 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552940, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:42:20.827327 15891 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:42:20.828958 15891 webserver.cc:460] Webserver started at http://127.31.61.129:41287/ using document root <none> and password file <none>
I20260812 16:42:20.829656 15891 fs_manager.cc:359] Metadata directory not provided
I20260812 16:42:20.829783 15891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:42:20.835700 15891 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:42:20.838706 15927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:20.840379 15891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 16:42:20.840586 15891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "66f36981f7b64841b0af4a437c616b33"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "a55eaa57ba7deb15382d8dd895e9f69c"
server_key_iv: "78484392b1eee6dce3086535ff7d8b2c"
server_key_version: "encryptionkey@0"
I20260812 16:42:20.841619 15891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:42:20.855374 15891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:42:20.860294 15891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:42:20.860571 15891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:42:20.861594 15891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:42:20.862977 15891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:42:20.863081 15891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:20.863186 15891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:42:20.863256 15891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:20.896351 15891 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:46077
I20260812 16:42:20.896418 16041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:46077 every 8 connection(s)
I20260812 16:42:20.897776 15891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:42:20.898967 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 15891
I20260812 16:42:20.899580 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:33607
--local_ip_for_outbound_sockets=127.31.61.130
--tserver_master_addrs=127.31.61.190:43821
--webserver_port=34911
--webserver_interface=127.31.61.130
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:42:20 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552940, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:42:20.921079 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:20.900888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:52447)
0812 16:42:20.902031 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:20.902083 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:20.903847 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:20.905342 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:20.905349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:20.905876 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:42:20.906414 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:20.906432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:20.908324 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:20.908327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:20.909529 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:20.909542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:20.910540 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:20.910548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:20.910797 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:20.911450 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:42:20.911472 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:42:20.913594 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:42:20.916770 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:20.916780 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:20.916795 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:20.917297 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:20.917716 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:20.917721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:20.917725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:20.917906 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:20.918625 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:42:20.918633 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:42:20.918938 (+ 305us) client_negotiation.cc:769] Sending connection context
0812 16:42:20.919187 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 16:42:20.919765 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":657,"threads_started":1}
I20260812 16:42:20.921079 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:20.900730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:52447 (local address 127.31.61.190:43821)
0812 16:42:20.901455 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:20.901472 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:20.902240 (+ 768us) server_negotiation.cc:407] Connection header received
0812 16:42:20.904418 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:20.904435 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:20.904856 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:20.905215 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:20.906654 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:20.908156 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:20.909672 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:20.910309 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:20.910477 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:20.913850 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:20.913858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:20.913872 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:20.913914 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:20.916628 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:20.917426 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:20.917433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:20.917438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:20.917603 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:20.918013 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:20.918018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:20.918020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:20.918477 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:20.918659 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:20.919460 (+ 801us) server_negotiation.cc:299] Negotiation successful
0812 16:42:20.919917 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":324,"threads_started":1}
I20260812 16:42:20.923874 16042 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43821
I20260812 16:42:20.924192 16042 heartbeater.cc:464] Registering TS with master...
I20260812 16:42:20.924906 16042 heartbeater.cc:511] Master 127.31.61.190:43821 requested a full tablet report, sending...
I20260812 16:42:20.926939 15851 ts_manager.cc:194] Registered new tserver with Master: 66f36981f7b64841b0af4a437c616b33 (127.31.61.129:46077)
I20260812 16:42:20.928524 15851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:52447
W20260812 16:42:21.239923 16046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:42:21.240253 16046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:42:21.240387 16046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:42:21.248261 16046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:42:21.248420 16046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:42:21.248574 16046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:42:21.259013 16046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:33607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=34911
--tserver_master_addrs=127.31.61.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:42:21.260492 16046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:42:21.261886 16046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:42:21.275703 16052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:21.275779 16051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:21.275926 16054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:42:21.276666 16046 server_base.cc:1034] running on GCE node
I20260812 16:42:21.277385 16046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:42:21.278450 16046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:42:21.279717 16046 hybrid_clock.cc:648] HybridClock initialized: now 1786552941279638 us; error 64 us; skew 500 ppm
I20260812 16:42:21.280069 16046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:42:21 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:42:21.283465 16046 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:42:21.285122 16046 webserver.cc:460] Webserver started at http://127.31.61.130:34911/ using document root <none> and password file <none>
I20260812 16:42:21.285825 16046 fs_manager.cc:359] Metadata directory not provided
I20260812 16:42:21.285961 16046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:42:21.291574 16046 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:42:21.294620 16061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:21.296106 16046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:42:21.296280 16046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "0ea790398f564aa8bb625575843b61c4"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "004817da9358c85d20fde203d351a4b8"
server_key_iv: "1379e857cb5de8f5b49564954d28090a"
server_key_version: "encryptionkey@0"
I20260812 16:42:21.297303 16046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:42:21.312685 16046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:42:21.316809 16046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:42:21.317071 16046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:42:21.318043 16046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:42:21.319417 16046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:42:21.319512 16046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:21.319648 16046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:42:21.319722 16046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:21.350845 16046 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:33607
I20260812 16:42:21.350914 16174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:33607 every 8 connection(s)
I20260812 16:42:21.352388 16046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:42:21.356187 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 16046
I20260812 16:42:21.356829 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:41667
--local_ip_for_outbound_sockets=127.31.61.131
--tserver_master_addrs=127.31.61.190:43821
--webserver_port=43031
--webserver_interface=127.31.61.131
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:42:21 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:42:21.374626 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.355333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:50931 (local address 127.31.61.190:43821)
0812 16:42:21.355623 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.355631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.357269 (+ 1638us) server_negotiation.cc:407] Connection header received
0812 16:42:21.359156 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.359161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.359250 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.359363 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:21.360993 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.361951 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.363761 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.364323 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.364454 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.368730 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:21.368735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:21.368739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:21.368775 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:21.371208 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:21.372158 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:21.372162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:21.372165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:21.372285 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:21.372815 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:21.372819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:21.372821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:21.373137 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:21.373259 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:21.374175 (+ 916us) server_negotiation.cc:299] Negotiation successful
0812 16:42:21.374381 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260812 16:42:21.376003 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.356034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:50931)
0812 16:42:21.357068 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:21.357122 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:21.358863 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:21.359658 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:21.359669 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:21.360218 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:42:21.360803 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.360822 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.362146 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.362151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.363578 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.363592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.364609 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.364619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.364924 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.365666 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:42:21.365700 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:42:21.368442 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:42:21.371362 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:21.371372 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:21.371392 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:21.372026 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:21.372411 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:21.372416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:21.372419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:21.372671 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:21.373294 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:42:21.373303 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:42:21.373782 (+ 479us) client_negotiation.cc:769] Sending connection context
0812 16:42:21.374087 (+ 305us) client_negotiation.cc:240] Negotiation successful
0812 16:42:21.374611 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":353,"threads_started":1}
I20260812 16:42:21.377502 16175 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43821
I20260812 16:42:21.377851 16175 heartbeater.cc:464] Registering TS with master...
I20260812 16:42:21.378788 16175 heartbeater.cc:511] Master 127.31.61.190:43821 requested a full tablet report, sending...
I20260812 16:42:21.380525 15851 ts_manager.cc:194] Registered new tserver with Master: 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
I20260812 16:42:21.381680 15851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:50931
W20260812 16:42:21.692155 16178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:42:21.692452 16178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:42:21.692548 16178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:42:21.699851 16178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:42:21.699977 16178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:42:21.700090 16178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:42:21.709967 16178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:41667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=43031
--tserver_master_addrs=127.31.61.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:42:21.711251 16178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:42:21.712615 16178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:42:21.726114 16186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:21.726104 16184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:21.726408 16183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:42:21.726965 16178 server_base.cc:1034] running on GCE node
I20260812 16:42:21.727572 16178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:42:21.728579 16178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:42:21.729774 16178 hybrid_clock.cc:648] HybridClock initialized: now 1786552941729728 us; error 35 us; skew 500 ppm
I20260812 16:42:21.730091 16178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:42:21 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:42:21.733237 16178 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:42:21.734692 16178 webserver.cc:460] Webserver started at http://127.31.61.131:43031/ using document root <none> and password file <none>
I20260812 16:42:21.735365 16178 fs_manager.cc:359] Metadata directory not provided
I20260812 16:42:21.735499 16178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:42:21.741137 16178 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:42:21.744067 16193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:21.745711 16178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 16:42:21.745887 16178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "bc022496702b4488b578fa142036a618"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "35aa76a3432f66f82a50948f02018aec"
server_key_iv: "cdae57920267795bae92124a16f51508"
server_key_version: "encryptionkey@0"
I20260812 16:42:21.747026 16178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:42:21.766450 16178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:42:21.770731 16178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:42:21.770998 16178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:42:21.772071 16178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:42:21.773608 16178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:42:21.773707 16178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:21.773818 16178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:42:21.773898 16178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:21.805294 16178 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:41667
I20260812 16:42:21.805325 16305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:41667 every 8 connection(s)
I20260812 16:42:21.806758 16178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:42:21.815899 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 16178
Aug 12 16:42:21 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552941, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:42:21.829021 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.809402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:35005 (local address 127.31.61.190:43821)
0812 16:42:21.809610 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.809615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.810726 (+ 1111us) server_negotiation.cc:407] Connection header received
0812 16:42:21.812645 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.812650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.812736 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.812823 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:21.814381 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.815126 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.817242 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.817913 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.817991 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.822235 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:21.822242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:21.822247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:21.822290 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:21.825678 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:21.826440 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:21.826445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:21.826449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:21.826603 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:21.827155 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:21.827159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:21.827161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:21.827447 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:21.827568 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:21.828521 (+ 953us) server_negotiation.cc:299] Negotiation successful
0812 16:42:21.828727 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 16:42:21.830343 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.809811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:35005)
0812 16:42:21.810548 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:21.810588 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:21.812362 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:21.813093 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:21.813102 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:21.813637 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:42:21.814206 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.814226 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.815289 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.815293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.817058 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.817073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.818321 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.818330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.818636 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.819371 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:42:21.819411 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:42:21.822008 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:42:21.825834 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:21.825842 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:21.825857 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:21.826303 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:21.826723 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:21.826728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:21.826731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:21.827022 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:21.827573 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:42:21.827582 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:42:21.828083 (+ 501us) client_negotiation.cc:769] Sending connection context
0812 16:42:21.828386 (+ 303us) client_negotiation.cc:240] Negotiation successful
0812 16:42:21.828927 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":287,"threads_started":1}
I20260812 16:42:21.831820 16307 heartbeater.cc:347] Connected to a master server at 127.31.61.190:43821
I20260812 16:42:21.832289 16307 heartbeater.cc:464] Registering TS with master...
I20260812 16:42:21.832863 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.817002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.31.61.190:43821)
0812 16:42:21.817923 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.817927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.817944 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:42:21.817996 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.817999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.818069 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.818162 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:21.820182 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.821233 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.822955 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.823518 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.823655 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.826728 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:21.826735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:21.826739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:21.826775 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:21.830101 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:21.830911 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:21.830916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:21.830918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:21.831034 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:21.831401 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:21.831406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:21.831409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:21.831722 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:21.832005 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:21.832287 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 16:42:21.832508 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":546,"threads_started":1}
I20260812 16:42:21.833204 16307 heartbeater.cc:511] Master 127.31.61.190:43821 requested a full tablet report, sending...
I20260812 16:42:21.834903 15850 ts_manager.cc:194] Registered new tserver with Master: bc022496702b4488b578fa142036a618 (127.31.61.131:41667)
I20260812 16:42:21.835919 15850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:35005
I20260812 16:42:21.848080 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:42:21.855500 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.849077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.31.61.190:43821)
0812 16:42:21.849262 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.849266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.849569 (+ 303us) server_negotiation.cc:407] Connection header received
0812 16:42:21.849745 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.849749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.849891 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.850034 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:42:21.850396 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.851172 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.853103 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.853813 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.853936 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.854590 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:42:21.854962 (+ 372us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 16:42:21.855265 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38904: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 16:42:21.855662 16311 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38904: token expired
W20260812 16:42:21.856261 15767 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.61.190:43821)
I20260812 16:42:21.869400 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.856759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.31.61.190:43821)
0812 16:42:21.856923 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.856927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.856941 (+ 14us) server_negotiation.cc:407] Connection header received
0812 16:42:21.857053 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.857056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.857111 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.857183 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:21.858606 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.859327 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.860831 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.861243 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.861332 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.864166 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:21.864171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:21.864174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:21.864208 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:21.866790 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:21.867559 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:21.867562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:21.867565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:21.867742 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:21.868172 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:21.868176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:21.868178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:21.868523 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:21.868701 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:21.868969 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 16:42:21.869141 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:42:21.872856 15850 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:42:21.880455 15850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:42:21.902997 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.892208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:41667 (local address 127.0.0.1:38160)
0812 16:42:21.893598 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:21.893644 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:21.893861 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:21.894847 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:21.894851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:21.894879 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:42:21.895311 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.895324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.897203 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.897209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.899704 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.899719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.901892 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.901898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.902104 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.902221 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 16:42:21.902349 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 16:42:21.902556 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":3,"thread_start_us":1170,"threads_started":1}
I20260812 16:42:21.904148 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.892476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.31.61.131:41667)
0812 16:42:21.893190 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.893199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.893750 (+ 551us) server_negotiation.cc:407] Connection header received
0812 16:42:21.894116 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.894124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.894364 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.894521 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:42:21.895535 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.897052 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.899917 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.901363 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.901499 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.901736 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:21.903648 (+ 1912us) server_negotiation.cc:299] Negotiation successful
0812 16:42:21.903820 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":397,"threads_started":1}
I20260812 16:42:21.905162 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.892852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.31.61.129:46077)
0812 16:42:21.893422 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.893430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.893845 (+ 415us) server_negotiation.cc:407] Connection header received
0812 16:42:21.894116 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.894124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.894366 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.894559 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:42:21.895359 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.898189 (+ 2830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.901735 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.903252 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.903377 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.903632 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:21.904652 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0812 16:42:21.904822 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":366,"threads_started":1}
I20260812 16:42:21.905537 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.892365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:46077 (local address 127.0.0.1:50210)
0812 16:42:21.893670 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:21.893692 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:21.893861 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:21.894672 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:21.894679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:21.894706 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:42:21.895103 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.895118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.898365 (+ 3247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.898370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.901557 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.901572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.903781 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.903787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.903963 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.903996 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:42:21.904080 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 16:42:21.904208 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"thread_start_us":1891,"threads_started":1}
I20260812 16:42:21.908542 16241 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf3fc4cf24c44a2b571d4a235eeb730 (DEFAULT_TABLE table=test-table [id=ef650cc7591f48d083829f27e7b52e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:42:21.908879 15977 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf3fc4cf24c44a2b571d4a235eeb730 (DEFAULT_TABLE table=test-table [id=ef650cc7591f48d083829f27e7b52e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:42:21.910046 16241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf3fc4cf24c44a2b571d4a235eeb730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:42:21.910194 15977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf3fc4cf24c44a2b571d4a235eeb730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:42:21.911293 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.894184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:33607 (local address 127.0.0.1:38416)
0812 16:42:21.903325 (+ 9141us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:21.903345 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:21.903456 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:21.904926 (+ 1470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:21.904932 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:21.904965 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:42:21.905127 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.905138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.906842 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.906848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.908619 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.908635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.910582 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.910588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.910752 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.910785 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 16:42:21.910881 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 16:42:21.911023 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9047,"thread_start_us":10201,"threads_started":1}
I20260812 16:42:21.911631 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.894852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.31.61.130:33607)
0812 16:42:21.895441 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.895450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.904322 (+ 8872us) server_negotiation.cc:407] Connection header received
0812 16:42:21.904448 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.904455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.904721 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.904930 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:42:21.905302 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.906686 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.908801 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.910159 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.910289 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.910464 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:21.910956 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 16:42:21.911124 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":305,"threads_started":1}
I20260812 16:42:21.915208 16110 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf3fc4cf24c44a2b571d4a235eeb730 (DEFAULT_TABLE table=test-table [id=ef650cc7591f48d083829f27e7b52e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:42:21.916764 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf3fc4cf24c44a2b571d4a235eeb730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:42:21.922573 16323 tablet_bootstrap.cc:492] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Bootstrap starting.
I20260812 16:42:21.925223 16324 tablet_bootstrap.cc:492] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: Bootstrap starting.
I20260812 16:42:21.927529 16323 tablet_bootstrap.cc:654] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Neither blocks nor log segments found. Creating new log.
I20260812 16:42:21.928890 16324 tablet_bootstrap.cc:654] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: Neither blocks nor log segments found. Creating new log.
I20260812 16:42:21.928866 16323 log.cc:824] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Log is configured to *not* fsync() on all Append() calls
I20260812 16:42:21.929991 16324 log.cc:824] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: Log is configured to *not* fsync() on all Append() calls
I20260812 16:42:21.931423 16042 heartbeater.cc:503] Master 127.31.61.190:43821 was elected leader, sending a full tablet report...
I20260812 16:42:21.931912 16326 tablet_bootstrap.cc:492] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: Bootstrap starting.
I20260812 16:42:21.932053 16323 tablet_bootstrap.cc:492] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: No bootstrap required, opened a new log
I20260812 16:42:21.932308 16323 ts_tablet_manager.cc:1397] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260812 16:42:21.935401 16326 tablet_bootstrap.cc:654] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: Neither blocks nor log segments found. Creating new log.
I20260812 16:42:21.935472 16324 tablet_bootstrap.cc:492] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: No bootstrap required, opened a new log
I20260812 16:42:21.935791 16324 ts_tablet_manager.cc:1397] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 16:42:21.936640 16326 log.cc:824] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: Log is configured to *not* fsync() on all Append() calls
I20260812 16:42:21.939622 16326 tablet_bootstrap.cc:492] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: No bootstrap required, opened a new log
I20260812 16:42:21.939888 16326 ts_tablet_manager.cc:1397] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260812 16:42:21.942761 16323 raft_consensus.cc:348] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:21.943223 16323 raft_consensus.cc:374] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:42:21.943311 16323 raft_consensus.cc:729] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f36981f7b64841b0af4a437c616b33, State: Initialized, Role: FOLLOWER
I20260812 16:42:21.943136 16324 raft_consensus.cc:348] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:21.943529 16324 raft_consensus.cc:374] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:42:21.943677 16324 raft_consensus.cc:729] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc022496702b4488b578fa142036a618, State: Initialized, Role: FOLLOWER
I20260812 16:42:21.944046 16323 consensus_queue.cc:260] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:21.944324 16324 consensus_queue.cc:260] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:21.946192 16323 ts_tablet_manager.cc:1428] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260812 16:42:21.947078 16307 heartbeater.cc:503] Master 127.31.61.190:43821 was elected leader, sending a full tablet report...
I20260812 16:42:21.947788 16324 ts_tablet_manager.cc:1428] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260812 16:42:21.950630 16326 raft_consensus.cc:348] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:21.951047 16326 raft_consensus.cc:374] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:42:21.951157 16326 raft_consensus.cc:729] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea790398f564aa8bb625575843b61c4, State: Initialized, Role: FOLLOWER
I20260812 16:42:21.951812 16326 consensus_queue.cc:260] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:21.952823 16175 heartbeater.cc:503] Master 127.31.61.190:43821 was elected leader, sending a full tablet report...
I20260812 16:42:21.953341 16326 ts_tablet_manager.cc:1428] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260812 16:42:21.960431 16329 raft_consensus.cc:482] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:42:21.960752 16329 raft_consensus.cc:504] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:21.962590 16329 leader_election.cc:288] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc022496702b4488b578fa142036a618 (127.31.61.131:41667), 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
I20260812 16:42:21.970407 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.963109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:46949 (local address 127.31.61.131:41667)
0812 16:42:21.963297 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.963302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.963700 (+ 398us) server_negotiation.cc:407] Connection header received
0812 16:42:21.963923 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.963930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.964010 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.964109 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:42:21.964838 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.965866 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.968454 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.969568 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.969745 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.969823 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:21.970067 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 16:42:21.970198 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:42:21.970595 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.962908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:41667 (local address 127.31.61.129:46949)
0812 16:42:21.963566 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:21.963586 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:21.963768 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:21.964168 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:21.964174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:21.964206 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:42:21.964677 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.964690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.966039 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.966046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.968284 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.968300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.969795 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.969799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.969919 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.969945 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 16:42:21.970016 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 16:42:21.970141 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":3,"thread_start_us":303,"threads_started":1}
I20260812 16:42:21.970959 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.963886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:52769 (local address 127.31.61.130:33607)
0812 16:42:21.964109 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.964116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.964731 (+ 615us) server_negotiation.cc:407] Connection header received
0812 16:42:21.964952 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.964958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.965039 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.965145 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:42:21.965755 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.966611 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.968867 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.969988 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.970165 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.970227 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:21.970645 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 16:42:21.970779 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 16:42:21.971547 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.963730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:33607 (local address 127.31.61.129:52769)
0812 16:42:21.964630 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:21.964645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:21.964761 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:21.965190 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:21.965194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:21.965219 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:42:21.965602 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.965611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.966742 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.966748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.968723 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:21.968734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:21.970235 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:21.970241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:21.970435 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:21.970466 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 16:42:21.971185 (+ 719us) client_negotiation.cc:240] Negotiation successful
0812 16:42:21.971282 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":84,"thread_start_us":579,"threads_started":1}
I20260812 16:42:21.971665 16261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf3fc4cf24c44a2b571d4a235eeb730" candidate_uuid: "66f36981f7b64841b0af4a437c616b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc022496702b4488b578fa142036a618" is_pre_election: true
I20260812 16:42:21.972203 16261 raft_consensus.cc:2454] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f36981f7b64841b0af4a437c616b33 in term 0.
I20260812 16:42:21.972404 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf3fc4cf24c44a2b571d4a235eeb730" candidate_uuid: "66f36981f7b64841b0af4a437c616b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea790398f564aa8bb625575843b61c4" is_pre_election: true
I20260812 16:42:21.972940 16130 raft_consensus.cc:2454] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f36981f7b64841b0af4a437c616b33 in term 0.
I20260812 16:42:21.972991 15929 leader_election.cc:302] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f36981f7b64841b0af4a437c616b33, bc022496702b4488b578fa142036a618; no voters:
I20260812 16:42:21.973452 16329 raft_consensus.cc:2781] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:42:21.973627 16329 raft_consensus.cc:482] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:42:21.973747 16329 raft_consensus.cc:3037] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:42:21.975979 16329 raft_consensus.cc:504] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:21.976548 16329 leader_election.cc:288] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 1 election: Requested vote from peers bc022496702b4488b578fa142036a618 (127.31.61.131:41667), 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
I20260812 16:42:21.977059 16261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf3fc4cf24c44a2b571d4a235eeb730" candidate_uuid: "66f36981f7b64841b0af4a437c616b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc022496702b4488b578fa142036a618"
I20260812 16:42:21.977077 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf3fc4cf24c44a2b571d4a235eeb730" candidate_uuid: "66f36981f7b64841b0af4a437c616b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea790398f564aa8bb625575843b61c4"
I20260812 16:42:21.977267 16130 raft_consensus.cc:3037] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:42:21.977268 16261 raft_consensus.cc:3037] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:42:21.979514 16130 raft_consensus.cc:2454] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f36981f7b64841b0af4a437c616b33 in term 1.
I20260812 16:42:21.979542 16261 raft_consensus.cc:2454] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f36981f7b64841b0af4a437c616b33 in term 1.
I20260812 16:42:21.979962 15929 leader_election.cc:302] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f36981f7b64841b0af4a437c616b33, bc022496702b4488b578fa142036a618; no voters:
I20260812 16:42:21.980258 16329 raft_consensus.cc:2781] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:42:21.980712 16329 raft_consensus.cc:686] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 LEADER]: Becoming Leader. State: Replica: 66f36981f7b64841b0af4a437c616b33, State: Running, Role: LEADER
I20260812 16:42:21.981184 16329 consensus_queue.cc:237] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:21.985790 15851 catalog_manager.cc:5495] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f36981f7b64841b0af4a437c616b33 (127.31.61.129). New cstate: current_term: 1 leader_uuid: "66f36981f7b64841b0af4a437c616b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } health_report { overall_health: UNKNOWN } } }
I20260812 16:42:22.004575 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:21.999274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.31.61.129:46077)
0812 16:42:21.999467 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:21.999471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:21.999488 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:42:21.999580 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:21.999584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:21.999719 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:21.999839 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:42:22.000220 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:22.000959 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:22.002444 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:22.002843 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:22.002934 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:22.003449 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:42:22.004000 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:42:22.004117 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:22.004258 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 16:42:22.004326 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:42:22.012684 16130 raft_consensus.cc:1261] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [term 1 FOLLOWER]: Refusing update from remote peer 66f36981f7b64841b0af4a437c616b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:42:22.012704 16261 raft_consensus.cc:1261] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [term 1 FOLLOWER]: Refusing update from remote peer 66f36981f7b64841b0af4a437c616b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:42:22.013710 16329 consensus_queue.cc:1035] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:42:22.014025 16334 consensus_queue.cc:1035] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:42:22.039670 15851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:42:22.040001 15851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:42:22.043854 15851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Sending DeleteTablet for 3 replicas of tablet 9cf3fc4cf24c44a2b571d4a235eeb730
I20260812 16:42:22.045047 16241 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf3fc4cf24c44a2b571d4a235eeb730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:42:22 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:38160
I20260812 16:42:22.045083 15977 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf3fc4cf24c44a2b571d4a235eeb730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:42:22 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:50210
I20260812 16:42:22.045161 16110 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf3fc4cf24c44a2b571d4a235eeb730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:42:22 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:38416
I20260812 16:42:22.045897 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 15891
I20260812 16:42:22.059134 16348 tablet_replica.cc:330] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: stopping tablet replica
I20260812 16:42:22.060066 16347 tablet_replica.cc:330] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: stopping tablet replica
I20260812 16:42:22.060542 16347 raft_consensus.cc:2229] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:42:22.061273 16347 raft_consensus.cc:2258] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:42:22.061897 16348 raft_consensus.cc:2229] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:42:22.062414 16348 raft_consensus.cc:2258] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:42:22.066164 16348 ts_tablet_manager.cc:1905] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:42:22.066499 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.046704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.31.61.129:46077)
0812 16:42:22.047169 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:22.047174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:22.047187 (+ 13us) server_negotiation.cc:407] Connection header received
0812 16:42:22.047229 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:22.047232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:22.047288 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:22.047370 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:22.049330 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:22.050641 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:22.052814 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:22.053643 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:22.053764 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:22.057453 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:22.057472 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:22.057499 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:22.057555 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:22.061808 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:22.062883 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:22.062891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:22.062894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:22.063079 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:22.063546 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:22.063550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:22.063554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:22.064057 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:22.065649 (+ 1592us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:22.065866 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 16:42:22.066108 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370}
I20260812 16:42:22.067245 16037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:42:22.068898 16346 tablet_replica.cc:330] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: stopping tablet replica
I20260812 16:42:22.069341 16346 raft_consensus.cc:2229] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:42:22.069828 16346 raft_consensus.cc:2258] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:42:22.072335 16346 ts_tablet_manager.cc:1905] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:42:22.073803 16348 ts_tablet_manager.cc:1918] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:42:22.073961 16348 log.cc:1196] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/9cf3fc4cf24c44a2b571d4a235eeb730
I20260812 16:42:22.074457 16348 ts_tablet_manager.cc:1939] T 9cf3fc4cf24c44a2b571d4a235eeb730 P bc022496702b4488b578fa142036a618: Deleting consensus metadata
I20260812 16:42:22.076300 15835 catalog_manager.cc:4841] TS bc022496702b4488b578fa142036a618 (127.31.61.131:41667): tablet 9cf3fc4cf24c44a2b571d4a235eeb730 (table test-table [id=ef650cc7591f48d083829f27e7b52e2f]) successfully deleted
I20260812 16:42:22.078434 16346 ts_tablet_manager.cc:1918] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:42:22.078586 16346 log.cc:1196] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/9cf3fc4cf24c44a2b571d4a235eeb730
I20260812 16:42:22.079044 16346 ts_tablet_manager.cc:1939] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 0ea790398f564aa8bb625575843b61c4: Deleting consensus metadata
I20260812 16:42:22.080493 15835 catalog_manager.cc:4841] TS 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607): tablet 9cf3fc4cf24c44a2b571d4a235eeb730 (table test-table [id=ef650cc7591f48d083829f27e7b52e2f]) successfully deleted
I20260812 16:42:22.069223 16347 ts_tablet_manager.cc:1905] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:42:22.164842 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 15891
W20260812 16:42:22.181627 15836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.61.129:46077 (error 108)
I20260812 16:42:22.181734 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 16046
I20260812 16:42:22.183485 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.183045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:46077 (local address 127.0.0.1:50234)
0812 16:42:22.183210 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:22.183314 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 16:42:22.183738 15836 catalog_manager.cc:4584] TS 66f36981f7b64841b0af4a437c616b33 (127.31.61.129:46077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cf3fc4cf24c44a2b571d4a235eeb730: Network error: Client connection negotiation failed: client connection to 127.31.61.129:46077: connect: Connection refused (error 111)
I20260812 16:42:22.195010 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.182476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.31.61.130:33607)
0812 16:42:22.182663 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:22.182668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:22.182740 (+ 72us) server_negotiation.cc:407] Connection header received
0812 16:42:22.182933 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:22.182938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:22.183014 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:22.183099 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:22.184596 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:22.185349 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:22.186624 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:22.187051 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:22.187161 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:22.189731 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:22.189745 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:22.189759 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:22.189791 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:22.192370 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:22.193025 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:22.193032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:22.193035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:22.193154 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:22.193523 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:22.193527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:22.193529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:22.193891 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:22.194180 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:22.194478 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 16:42:22.194685 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:42:22.195679 16170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:42:22.237473 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.236919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:46077 (local address 127.0.0.1:50244)
0812 16:42:22.237182 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:22.237309 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260812 16:42:22.238729 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.238275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:46077 (local address 127.0.0.1:50250)
0812 16:42:22.238497 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:22.238584 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 16:42:22.284785 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 16046
I20260812 16:42:22.299499 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.298957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:46077 (local address 127.0.0.1:50256)
0812 16:42:22.299209 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:22.299299 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260812 16:42:22.299499 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 16178
I20260812 16:42:22.300585 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.300225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:46077 (local address 127.0.0.1:50266)
0812 16:42:22.300362 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:22.300453 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 16:42:22.316540 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.300328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.31.61.131:41667)
0812 16:42:22.300504 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:22.300509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:22.300531 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:42:22.300637 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:22.300643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:22.300712 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:22.300796 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:22.302132 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:22.303180 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:22.304617 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:22.305238 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:22.305340 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:22.308998 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:22.309018 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:22.309041 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:22.309085 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:22.313011 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:22.313880 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:22.313891 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:22.313895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:22.314108 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:22.314684 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:22.314690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:22.314694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:22.315145 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:22.315468 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:22.315947 (+ 479us) server_negotiation.cc:299] Negotiation successful
0812 16:42:22.316166 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 16:42:22.317435 16301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:42:22.381250 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.380779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:46077 (local address 127.0.0.1:50276)
0812 16:42:22.380983 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:22.381102 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 16:42:22.382427 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:22.381980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:46077 (local address 127.0.0.1:50284)
0812 16:42:22.382201 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:22.382291 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260812 16:42:22.412781 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 16178
I20260812 16:42:22.432155 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 15816
I20260812 16:42:22.433476 15884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:42:22.549027 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 15816
I20260812 16:42:42.568490 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43821
--webserver_interface=127.31.61.190
--webserver_port=36071
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:42:42.915819 16354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:42:42.916112 16354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:42:42.916201 16354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:42:42.924501 16354 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 16:42:42.924612 16354 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 16:42:42.924650 16354 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 16:42:42.924693 16354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:42:42.924746 16354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:42:42.924813 16354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:42:42.924856 16354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:42:42.924909 16354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:42:42.935302 16354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=36071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:42:42.936656 16354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:42:42.937947 16354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:42:42.949749 16361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:42.949724 16360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:42.949980 16363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:42:42.950462 16354 server_base.cc:1034] running on GCE node
I20260812 16:42:42.951239 16354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:42:42.952689 16354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:42:42.953962 16354 hybrid_clock.cc:648] HybridClock initialized: now 1786552962953889 us; error 46 us; skew 500 ppm
I20260812 16:42:42.954268 16354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:42:42 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552962, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:42:42.957496 16354 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:42:42.958998 16354 webserver.cc:460] Webserver started at http://127.31.61.190:36071/ using document root <none> and password file <none>
I20260812 16:42:42.959743 16354 fs_manager.cc:359] Metadata directory not provided
I20260812 16:42:42.959838 16354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:42:42.965564 16354 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:42:42.968640 16370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:42.970147 16354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:42:42.970333 16354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "2c4b57d6d4944e209c4da660438ce36f"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "bd9387c364fcb497ba0f90c5ee4110bb"
server_key_iv: "608d72f22c746bced61fda430399998d"
server_key_version: "encryptionkey@0"
I20260812 16:42:42.971299 16354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:42:42.996752 16354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:42:43.001256 16354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:42:43.001547 16354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:42:43.020479 16354 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:43821
I20260812 16:42:43.020515 16423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:43821 every 8 connection(s)
I20260812 16:42:43.021987 16354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:42:43.025024 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 16354
I20260812 16:42:43.026053 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:46077
--local_ip_for_outbound_sockets=127.31.61.129
--tserver_master_addrs=127.31.61.190:43821
--webserver_port=41287
--webserver_interface=127.31.61.129
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:42:43.030750 16424 sys_catalog.cc:263] Verifying existing consensus state
I20260812 16:42:43.035238 16424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Bootstrap starting.
I20260812 16:42:43.055989 16424 log.cc:824] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:42:43.072705 16424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:42:43.073235 16424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Bootstrap complete.
I20260812 16:42:43.082480 16424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:42:43.083389 16424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4b57d6d4944e209c4da660438ce36f, State: Initialized, Role: FOLLOWER
I20260812 16:42:43.084079 16424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:42:43.084303 16424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:42:43.084430 16424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:42:43.084594 16424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 2 FOLLOWER]: Advancing to term 3
I20260812 16:42:43.088531 16424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:42:43.088994 16424 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4b57d6d4944e209c4da660438ce36f; no voters:
I20260812 16:42:43.089646 16424 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 16:42:43.089787 16428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 3 FOLLOWER]: Leader election won for term 3
I20260812 16:42:43.090878 16428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [term 3 LEADER]: Becoming Leader. State: Replica: 2c4b57d6d4944e209c4da660438ce36f, State: Running, Role: LEADER
I20260812 16:42:43.091426 16428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } }
I20260812 16:42:43.091817 16424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:42:43.094017 16430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4b57d6d4944e209c4da660438ce36f. Latest consensus state: current_term: 3 leader_uuid: "2c4b57d6d4944e209c4da660438ce36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } } }
I20260812 16:42:43.094048 16429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2c4b57d6d4944e209c4da660438ce36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b57d6d4944e209c4da660438ce36f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 43821 } } }
I20260812 16:42:43.094447 16430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: This master's current role is: LEADER
I20260812 16:42:43.094453 16429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f [sys.catalog]: This master's current role is: LEADER
I20260812 16:42:43.105010 16436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:42:43.113830 16436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:42:43.115321 16436 catalog_manager.cc:1259] Loaded cluster ID: 9f7fe40e2f8b4c3096849149814ec379
I20260812 16:42:43.115469 16436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:42:43.119819 16436 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:42:43.121562 16436 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Loaded TSK: 0
I20260812 16:42:43.121730 16436 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c4b57d6d4944e209c4da660438ce36f: Loaded TSK: 1
F20260812 16:42:43.122874 16436 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f950daa79ad google::LogMessage::Fail() at ??:0
@ 0x7f950daaa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f950daa74de google::LogMessage::Flush() at ??:0
@ 0x7f950daa8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f951180b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f95117fa73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f95234da1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f95234d9783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f95235652a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f95235650c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f95235af5e9 std::function<>::operator()() at ??:0
@ 0x7f95234ddcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f95234dbc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9523563610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9523563439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f95106f3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f95106f691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f95106f6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f95106d1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f95139ab6db start_thread at ??:0
@ 0x7f950c94161f clone at ??:0
W20260812 16:42:43.424010 16426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:42:43.424294 16426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:42:43.424394 16426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:42:43.431757 16426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:42:43.431874 16426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:42:43.431996 16426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:42:43.441946 16426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:46077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=41287
--tserver_master_addrs=127.31.61.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:42:43.443234 16426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:42:43.444551 16426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:42:43.457645 16473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:43.457604 16475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:43.457623 16472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:42:43.458290 16426 server_base.cc:1034] running on GCE node
I20260812 16:42:43.459124 16426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:42:43.460183 16426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:42:43.461380 16426 hybrid_clock.cc:648] HybridClock initialized: now 1786552963461332 us; error 37 us; skew 500 ppm
I20260812 16:42:43.461710 16426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:42:43 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552963, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:42:43.464818 16426 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:42:43.466274 16426 webserver.cc:460] Webserver started at http://127.31.61.129:41287/ using document root <none> and password file <none>
I20260812 16:42:43.466950 16426 fs_manager.cc:359] Metadata directory not provided
I20260812 16:42:43.467075 16426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:42:43.472750 16426 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 16:42:43.475713 16482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:43.477046 16426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:42:43.477205 16426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "66f36981f7b64841b0af4a437c616b33"
format_stamp: "Formatted at 2026-08-12 16:41:57 on dist-test-slave-f7th"
server_key: "a55eaa57ba7deb15382d8dd895e9f69c"
server_key_iv: "78484392b1eee6dce3086535ff7d8b2c"
server_key_version: "encryptionkey@0"
I20260812 16:42:43.478164 16426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:42:43.499393 16426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:42:43.503400 16426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:42:43.503703 16426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:42:43.504705 16426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:42:43.506560 16489 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 16:42:43.512017 16426 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 16:42:43.512133 16426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260812 16:42:43.512251 16426 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 16:42:43.515316 16426 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 16:42:43.515400 16426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260812 16:42:43.515787 16489 tablet_bootstrap.cc:492] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Bootstrap starting.
I20260812 16:42:43.524575 16489 log.cc:824] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Log is configured to *not* fsync() on all Append() calls
I20260812 16:42:43.542030 16489 tablet_bootstrap.cc:492] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 16:42:43.542964 16489 tablet_bootstrap.cc:492] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Bootstrap complete.
I20260812 16:42:43.543610 16489 ts_tablet_manager.cc:1397] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.010s
I20260812 16:42:43.549710 16426 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:46077
I20260812 16:42:43.549747 16598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:46077 every 8 connection(s)
I20260812 16:42:43.551283 16426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:42:43.554049 16489 raft_consensus.cc:348] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:43.555054 16489 raft_consensus.cc:729] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f36981f7b64841b0af4a437c616b33, State: Initialized, Role: FOLLOWER
I20260812 16:42:43.555737 16489 consensus_queue.cc:260] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:43.556257 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 16426
I20260812 16:42:43.556730 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:43.554243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:35949)
0812 16:42:43.555146 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:43.555459 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":400,"threads_started":1}
I20260812 16:42:43.557109 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:33607
--local_ip_for_outbound_sockets=127.31.61.130
--tserver_master_addrs=127.31.61.190:43821
--webserver_port=34911
--webserver_interface=127.31.61.130
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:42:43.557293 16489 ts_tablet_manager.cc:1428] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
W20260812 16:42:43.558058 16599 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:43821 (0 consecutive failures): Network error: Failed to ping master at 127.31.61.190:43821: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
I20260812 16:42:43.558930 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:43.558579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:44673)
0812 16:42:43.558732 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:43.558809 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 16:42:43.559791 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:43.559410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:41783)
0812 16:42:43.559546 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:43.559653 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 16:42:43.560060 16599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:42:43.883235 16603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:42:43.883548 16603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:42:43.883701 16603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:42:43.891245 16603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:42:43.891368 16603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:42:43.891532 16603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:42:43.901340 16603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:33607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=34911
--tserver_master_addrs=127.31.61.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:42:43.902630 16603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:42:43.903955 16603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:42:43.916647 16608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:43.916673 16611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:43.916599 16609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:42:43.917479 16603 server_base.cc:1034] running on GCE node
I20260812 16:42:43.918077 16603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:42:43.919003 16603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:42:43.920214 16603 hybrid_clock.cc:648] HybridClock initialized: now 1786552963920148 us; error 46 us; skew 500 ppm
I20260812 16:42:43.920573 16603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:42:43 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552963, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:42:43.923739 16603 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:42:43.926622 16603 webserver.cc:460] Webserver started at http://127.31.61.130:34911/ using document root <none> and password file <none>
I20260812 16:42:43.927316 16603 fs_manager.cc:359] Metadata directory not provided
I20260812 16:42:43.927429 16603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:42:43.932811 16603 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260812 16:42:43.935458 16618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:43.936940 16603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:42:43.937105 16603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "0ea790398f564aa8bb625575843b61c4"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "004817da9358c85d20fde203d351a4b8"
server_key_iv: "1379e857cb5de8f5b49564954d28090a"
server_key_version: "encryptionkey@0"
I20260812 16:42:43.938094 16603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:42:43.955632 16603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:42:43.960237 16603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:42:43.960503 16603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:42:43.961588 16603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:42:43.963186 16603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:42:43.963287 16603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:43.963438 16603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:42:43.963519 16603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:43.995023 16603 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:33607
I20260812 16:42:43.995080 16732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:33607 every 8 connection(s)
I20260812 16:42:43.996510 16603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:42:44.001432 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:43.999468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:36727)
0812 16:42:44.000316 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:44.000556 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":337,"threads_started":1}
W20260812 16:42:44.002146 16733 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:43821 (0 consecutive failures): Network error: Failed to ping master at 127.31.61.190:43821: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
I20260812 16:42:44.002578 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 16603
I20260812 16:42:44.003202 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:44.002799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:45719)
0812 16:42:44.002977 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:44.003066 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 16:42:44.003273 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:41667
--local_ip_for_outbound_sockets=127.31.61.131
--tserver_master_addrs=127.31.61.190:43821
--webserver_port=43031
--webserver_interface=127.31.61.131
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:42:44.004259 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:44.003873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:55257)
0812 16:42:44.004029 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:44.004124 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 16:42:44.004560 16733 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 16:42:44.341601 16736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:42:44.341902 16736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:42:44.342026 16736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:42:44.349673 16736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:42:44.349802 16736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:42:44.349956 16736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:42:44.360459 16736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:41667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=43031
--tserver_master_addrs=127.31.61.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:42:44.361871 16736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:42:44.363547 16736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:42:44.377137 16742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:44.377159 16744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:42:44.377187 16741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:42:44.381426 16736 server_base.cc:1034] running on GCE node
I20260812 16:42:44.382145 16736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:42:44.383180 16736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:42:44.384482 16736 hybrid_clock.cc:648] HybridClock initialized: now 1786552964384410 us; error 58 us; skew 500 ppm
I20260812 16:42:44.384831 16736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:42:44 dist-test-slave-f7th krb5kdc[15252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552964, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:42:44.388412 16736 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:42:44.390215 16736 webserver.cc:460] Webserver started at http://127.31.61.131:43031/ using document root <none> and password file <none>
I20260812 16:42:44.391109 16736 fs_manager.cc:359] Metadata directory not provided
I20260812 16:42:44.391254 16736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:42:44.396981 16736 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 16:42:44.399912 16751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:44.401499 16736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:42:44.401688 16736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "bc022496702b4488b578fa142036a618"
format_stamp: "Formatted at 2026-08-12 16:41:58 on dist-test-slave-f7th"
server_key: "35aa76a3432f66f82a50948f02018aec"
server_key_iv: "cdae57920267795bae92124a16f51508"
server_key_version: "encryptionkey@0"
I20260812 16:42:44.402783 16736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:42:44.416893 16736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:42:44.421448 16736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:42:44.421730 16736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:42:44.422760 16736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:42:44.424230 16736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:42:44.424330 16736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:44.424436 16736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:42:44.424504 16736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:42:44.457098 16736 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:41667
I20260812 16:42:44.457160 16864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:41667 every 8 connection(s)
I20260812 16:42:44.458493 16736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:42:44.459182 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 16736
I20260812 16:42:44.463559 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:44.461741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:51205)
0812 16:42:44.462478 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:44.462714 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":289,"threads_started":1}
W20260812 16:42:44.464275 16865 heartbeater.cc:644] Failed to heartbeat to 127.31.61.190:43821 (0 consecutive failures): Network error: Failed to ping master at 127.31.61.190:43821: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
I20260812 16:42:44.465163 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:44.464828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:52951)
0812 16:42:44.464990 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:44.465059 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 16:42:44.465998 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:44.465685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:44169)
0812 16:42:44.465804 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:44.465878 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 16:42:44.466223 16865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 16:42:44.561496 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:44.560616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:52951)
0812 16:42:44.561112 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:44.561234 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":214,"threads_started":1}
I20260812 16:42:45.006094 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:45.005216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:51901)
0812 16:42:45.005727 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:45.005842 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":260,"threads_started":1}
I20260812 16:42:45.163630 16871 raft_consensus.cc:482] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:42:45.163969 16871 raft_consensus.cc:504] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:45.165555 16871 leader_election.cc:288] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc022496702b4488b578fa142036a618 (127.31.61.131:41667), 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
Aug 12 16:42:45 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552963, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
Aug 12 16:42:45 dist-test-slave-f7th krb5kdc[15252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552963, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:42:45.192986 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:45.166706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:33607 (local address 127.31.61.129:36547)
0812 16:42:45.167979 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:45.167999 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:45.168114 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:45.171824 (+ 3710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:45.171836 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:45.173050 (+ 1214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:42:45.174328 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:45.174364 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:45.176834 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:45.176841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:45.179396 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:45.179415 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:45.180926 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:45.180940 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:45.181354 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:45.182468 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:42:45.182505 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:42:45.185591 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:42:45.189134 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:45.189138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:45.189147 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:45.189760 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:45.190213 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:45.190217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:45.190219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:45.190447 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:45.191318 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:42:45.191323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:42:45.191627 (+ 304us) client_negotiation.cc:769] Sending connection context
0812 16:42:45.191965 (+ 338us) client_negotiation.cc:240] Negotiation successful
0812 16:42:45.192511 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":438,"spinlock_wait_cycles":53248,"thread_start_us":960,"threads_started":1}
I20260812 16:42:45.193038 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:45.167185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:36547 (local address 127.31.61.130:33607)
0812 16:42:45.168662 (+ 1477us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:45.168675 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:45.168718 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:42:45.170821 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:45.170833 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:45.171434 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:45.171783 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:45.174626 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:45.176573 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:45.179594 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:45.180648 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:45.180849 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:45.185892 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:45.185921 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:45.185934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:45.185973 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:45.188976 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:45.189901 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:45.189908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:45.189914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:45.190091 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:45.190554 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:45.190558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:45.190560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:45.191066 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:45.191249 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:45.192220 (+ 971us) server_negotiation.cc:299] Negotiation successful
0812 16:42:45.192632 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1300,"thread_start_us":1118,"threads_started":1}
I20260812 16:42:45.193465 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:45.166184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:59765 (local address 127.31.61.131:41667)
0812 16:42:45.168244 (+ 2060us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:45.168261 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:45.168303 (+ 42us) server_negotiation.cc:407] Connection header received
0812 16:42:45.170445 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:45.170459 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:45.171202 (+ 743us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:45.171655 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:45.174626 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:45.176568 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:45.179593 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:45.180649 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:45.180849 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:45.185686 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:45.185700 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:45.185721 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:45.185770 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:45.188935 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:45.189903 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:45.189910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:45.189916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:45.190090 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:45.190528 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:45.190533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:45.190536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:45.191066 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:45.191220 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:45.192214 (+ 994us) server_negotiation.cc:299] Negotiation successful
0812 16:42:45.192632 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1880,"thread_start_us":1764,"threads_started":1}
I20260812 16:42:45.193035 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:45.165991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:41667 (local address 127.31.61.129:59765)
0812 16:42:45.166517 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:45.166565 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 16:42:45.167166 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:42:45.171824 (+ 4658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:42:45.171836 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:42:45.173050 (+ 1214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:42:45.174328 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:45.174364 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:45.176836 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:45.176841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:45.179400 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:42:45.179415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:42:45.180959 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:42:45.180967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:42:45.181354 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:45.182750 (+ 1396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:42:45.182774 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:42:45.185296 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:42:45.189117 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:45.189128 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:45.189144 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:45.189757 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:45.190213 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:42:45.190217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:42:45.190220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:42:45.190422 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:42:45.191223 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:42:45.191231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:42:45.191627 (+ 396us) client_negotiation.cc:769] Sending connection context
0812 16:42:45.191965 (+ 338us) client_negotiation.cc:240] Negotiation successful
0812 16:42:45.192659 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":472,"spinlock_wait_cycles":32000,"thread_start_us":262,"threads_started":1}
W20260812 16:42:45.196830 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
W20260812 16:42:45.197230 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc022496702b4488b578fa142036a618 (127.31.61.131:41667): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
I20260812 16:42:45.197365 16484 leader_election.cc:302] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f36981f7b64841b0af4a437c616b33; no voters: 0ea790398f564aa8bb625575843b61c4, bc022496702b4488b578fa142036a618
I20260812 16:42:45.197722 16871 raft_consensus.cc:2726] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:42:45.467726 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:45.466806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:46213)
0812 16:42:45.467281 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:45.467439 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":232,"threads_started":1}
I20260812 16:42:45.562901 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:45.562424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:40113)
0812 16:42:45.562648 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:45.562753 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260812 16:42:46.007934 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:46.007168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:60149)
0812 16:42:46.007671 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:46.007764 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":244,"threads_started":1}
I20260812 16:42:46.469278 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:46.468628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:55059)
0812 16:42:46.469040 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:46.469124 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":212,"threads_started":1}
I20260812 16:42:46.564780 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:46.563885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:50975)
0812 16:42:46.564443 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:46.564541 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":288,"threads_started":1}
I20260812 16:42:47.009640 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:47.008900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:54385)
0812 16:42:47.009366 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:47.009460 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":232,"threads_started":1}
I20260812 16:42:47.057837 16881 raft_consensus.cc:482] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:42:47.058022 16881 raft_consensus.cc:504] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:47.058672 16881 leader_election.cc:288] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc022496702b4488b578fa142036a618 (127.31.61.131:41667), 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
W20260812 16:42:47.059830 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
W20260812 16:42:47.060122 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc022496702b4488b578fa142036a618 (127.31.61.131:41667): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
I20260812 16:42:47.060221 16484 leader_election.cc:302] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f36981f7b64841b0af4a437c616b33; no voters: 0ea790398f564aa8bb625575843b61c4, bc022496702b4488b578fa142036a618
I20260812 16:42:47.060582 16881 raft_consensus.cc:2726] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:42:47.470997 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:47.470216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:35287)
0812 16:42:47.470734 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:47.470824 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":280,"threads_started":1}
I20260812 16:42:47.566646 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:47.565744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:45123)
0812 16:42:47.566287 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:47.566397 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":287,"threads_started":1}
I20260812 16:42:48.011577 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:48.010740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:58175)
0812 16:42:48.011255 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:48.011364 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":264,"threads_started":1}
I20260812 16:42:48.472712 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:48.471971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:35219)
0812 16:42:48.472404 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:48.472492 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":206,"threads_started":1}
I20260812 16:42:48.568640 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:48.567843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:47539)
0812 16:42:48.568352 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:48.568453 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":221,"threads_started":1}
I20260812 16:42:49.013540 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:49.012745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:38703)
0812 16:42:49.013265 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:49.013358 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":278,"threads_started":1}
I20260812 16:42:49.474490 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:49.473717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:43745)
0812 16:42:49.474228 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:49.474314 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":289,"threads_started":1}
I20260812 16:42:49.570308 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:49.569597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:36543)
0812 16:42:49.570059 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:49.570145 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260812 16:42:50.015476 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:50.014599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:54265)
0812 16:42:50.015194 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:50.015285 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":369,"threads_started":1}
I20260812 16:42:50.101308 16891 raft_consensus.cc:482] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:42:50.101540 16891 raft_consensus.cc:504] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:50.102159 16891 leader_election.cc:288] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc022496702b4488b578fa142036a618 (127.31.61.131:41667), 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
W20260812 16:42:50.103256 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc022496702b4488b578fa142036a618 (127.31.61.131:41667): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
W20260812 16:42:50.103487 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
I20260812 16:42:50.103621 16484 leader_election.cc:302] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f36981f7b64841b0af4a437c616b33; no voters: 0ea790398f564aa8bb625575843b61c4, bc022496702b4488b578fa142036a618
I20260812 16:42:50.103854 16891 raft_consensus.cc:2726] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:42:50.476204 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:50.475464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:41315)
0812 16:42:50.475937 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:50.476023 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":251,"threads_started":1}
I20260812 16:42:50.572362 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:50.571443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:37909)
0812 16:42:50.572040 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:50.572148 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":317,"threads_started":1}
I20260812 16:42:51.017530 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:51.016656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:37001)
0812 16:42:51.017241 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:51.017334 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":278,"threads_started":1}
I20260812 16:42:51.478181 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:51.477316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:38419)
0812 16:42:51.477878 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:51.477984 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":241,"threads_started":1}
I20260812 16:42:51.574198 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:51.573418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:44473)
0812 16:42:51.573928 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:51.574031 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":275,"threads_started":1}
I20260812 16:42:52.019373 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:52.018590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:38369)
0812 16:42:52.019067 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:52.019167 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":249,"threads_started":1}
I20260812 16:42:52.479944 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:52.479173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:33843)
0812 16:42:52.479654 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:52.479745 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":227,"threads_started":1}
I20260812 16:42:52.576041 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:52.575208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:50857)
0812 16:42:52.575769 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:52.575853 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":289,"threads_started":1}
I20260812 16:42:53.021325 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:53.020509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:33761)
0812 16:42:53.021036 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:53.021150 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":242,"threads_started":1}
I20260812 16:42:53.481787 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:53.480923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:47011)
0812 16:42:53.481532 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:53.481624 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":276,"threads_started":1}
I20260812 16:42:53.578038 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:53.577254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:46037)
0812 16:42:53.577765 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:53.577861 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":265,"threads_started":1}
I20260812 16:42:54.023046 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:54.022325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:55417)
0812 16:42:54.022779 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:54.022862 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":250,"threads_started":1}
I20260812 16:42:54.483740 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:54.482918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:48943)
0812 16:42:54.483401 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:54.483502 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":256,"threads_started":1}
I20260812 16:42:54.579949 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:54.579091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:55091)
0812 16:42:54.579591 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:54.579747 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260812 16:42:54.644205 16906 raft_consensus.cc:482] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:42:54.644436 16906 raft_consensus.cc:504] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:54.645241 16906 leader_election.cc:288] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc022496702b4488b578fa142036a618 (127.31.61.131:41667), 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
W20260812 16:42:54.646108 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
W20260812 16:42:54.646302 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc022496702b4488b578fa142036a618 (127.31.61.131:41667): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
I20260812 16:42:54.646368 16484 leader_election.cc:302] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f36981f7b64841b0af4a437c616b33; no voters: 0ea790398f564aa8bb625575843b61c4, bc022496702b4488b578fa142036a618
I20260812 16:42:54.646615 16906 raft_consensus.cc:2726] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:42:55.024883 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:55.024084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:43583)
0812 16:42:55.024615 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:55.024710 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":283,"threads_started":1}
I20260812 16:42:55.485526 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:55.484694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:40907)
0812 16:42:55.485235 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:55.485335 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":268,"threads_started":1}
I20260812 16:42:55.581998 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:55.581190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:56825)
0812 16:42:55.581711 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:55.581821 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":268,"threads_started":1}
I20260812 16:42:56.026737 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:56.025959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:42677)
0812 16:42:56.026430 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:56.026546 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":235,"threads_started":1}
I20260812 16:42:56.326987 16911 raft_consensus.cc:482] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:42:56.327188 16911 raft_consensus.cc:504] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc022496702b4488b578fa142036a618" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 41667 } } peers { permanent_uuid: "66f36981f7b64841b0af4a437c616b33" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 46077 } } peers { permanent_uuid: "0ea790398f564aa8bb625575843b61c4" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 33607 } }
I20260812 16:42:56.327901 16911 leader_election.cc:288] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc022496702b4488b578fa142036a618 (127.31.61.131:41667), 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607)
W20260812 16:42:56.328902 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc022496702b4488b578fa142036a618 (127.31.61.131:41667): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
W20260812 16:42:56.329113 16484 leader_election.cc:341] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ea790398f564aa8bb625575843b61c4 (127.31.61.130:33607): Not found: Tablet not found: 9cf3fc4cf24c44a2b571d4a235eeb730
I20260812 16:42:56.329209 16484 leader_election.cc:302] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f36981f7b64841b0af4a437c616b33; no voters: 0ea790398f564aa8bb625575843b61c4, bc022496702b4488b578fa142036a618
I20260812 16:42:56.329450 16911 raft_consensus.cc:2726] T 9cf3fc4cf24c44a2b571d4a235eeb730 P 66f36981f7b64841b0af4a437c616b33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 16:42:56.487288 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:56.486464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:52637)
0812 16:42:56.487013 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:56.487106 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":299,"threads_started":1}
I20260812 16:42:56.583948 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:56.583057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:60319)
0812 16:42:56.583558 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:56.583718 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":238,"threads_started":1}
I20260812 16:42:57.028478 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:57.027748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:47989)
0812 16:42:57.028223 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:57.028317 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":242,"threads_started":1}
I20260812 16:42:57.488945 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:57.488202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:44835)
0812 16:42:57.488682 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:57.488780 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":257,"threads_started":1}
I20260812 16:42:57.585778 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:57.584961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:33275)
0812 16:42:57.585437 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:57.585572 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260812 16:42:58.030097 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:58.029345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:46537)
0812 16:42:58.029815 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:58.029898 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":264,"threads_started":1}
I20260812 16:42:58.490639 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:58.489840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:54903)
0812 16:42:58.490338 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:58.490431 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":234,"threads_started":1}
I20260812 16:42:58.587772 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:58.587005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.129:53885)
0812 16:42:58.587453 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:58.587541 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":227,"threads_started":1}
I20260812 16:42:59.032163 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:59.031301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.130:60367)
0812 16:42:59.031868 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:59.031990 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":305,"threads_started":1}
I20260812 16:42:59.469823 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 16426
I20260812 16:42:59.489732 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:59.471008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.31.61.129:46077)
0812 16:42:59.471559 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:59.471569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:59.471626 (+ 57us) server_negotiation.cc:407] Connection header received
0812 16:42:59.471710 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:59.471721 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:59.472044 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:59.472233 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:59.473757 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:59.474733 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:59.476018 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:59.476602 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:59.476709 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:59.484152 (+ 7443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:59.484166 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:59.484181 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:59.484221 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:59.487217 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:59.487957 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:59.487964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:59.487967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:59.488112 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:59.488392 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:59.488395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:59.488397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:59.488759 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:59.488985 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:59.489261 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 16:42:59.489460 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":271,"threads_started":1}
I20260812 16:42:59.491010 16594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:42:59.495728 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:59.491746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:43821 (local address 127.31.61.131:56227)
0812 16:42:59.495343 (+ 3597us) negotiation.cc:107] Waiting for socket to connect
0812 16:42:59.495498 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.190:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3464,"thread_start_us":3327,"threads_started":1}
I20260812 16:42:59.580608 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 16426
I20260812 16:42:59.596643 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 16603
I20260812 16:42:59.610237 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:59.597422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.31.61.130:33607)
0812 16:42:59.597940 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:59.597944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:59.597964 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:42:59.598022 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:59.598026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:59.598091 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:59.598191 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:59.599545 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:59.600403 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:59.601647 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:59.602075 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:59.602168 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:59.604980 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:59.604986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:59.604991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:59.605025 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:59.607658 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:59.608338 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:59.608344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:59.608346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:59.608481 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:59.608822 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:59.608826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:59.608828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:59.609102 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:59.609374 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:59.609650 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 16:42:59.609866 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":270,"threads_started":1}
I20260812 16:42:59.611007 16728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:42:59.694208 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 16603
I20260812 16:42:59.709568 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 16736
I20260812 16:42:59.723316 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:42:59.710373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.31.61.131:41667)
0812 16:42:59.710929 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0812 16:42:59.710934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:42:59.710953 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:42:59.711008 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:42:59.711012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:42:59.711079 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:42:59.711170 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:42:59.712722 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:59.713549 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:59.714788 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:42:59.715220 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:42:59.715329 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:42:59.718069 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:42:59.718075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:42:59.718079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:42:59.718112 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:42:59.720842 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:59.721530 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:59.721534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:59.721537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:59.721672 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:42:59.722074 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:42:59.722078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:42:59.722080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:42:59.722366 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:42:59.722615 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 16:42:59.722851 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 16:42:59.723031 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":297,"threads_started":1}
I20260812 16:42:59.724067 16860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:42:59.807415 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 16736
2026-08-12T16:42:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64478 ms)
[----------] 30 tests from SecurityITest (303056 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:42:59 dist-test-slave-f7th krb5kdc[16930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:42:59 dist-test-slave-f7th krb5kdc[16930](info): set up 2 sockets
Aug 12 16:42:59 dist-test-slave-f7th krb5kdc[16930](info): commencing operation
krb5kdc: starting...
W20260812 16:43:00.847539 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:01 dist-test-slave-f7th krb5kdc[16930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:01.121568 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:34871
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:34871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:01.463183 16944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:01.463492 16944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:01.463651 16944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:01.471062 16944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:01.471185 16944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:01.471246 16944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:01.471325 16944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:01.471380 16944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:01.481529 16944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:34871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:01.482878 16944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:01.484277 16944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:01.495828 16949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:01.495761 16950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:01.495829 16952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:01.496960 16944 server_base.cc:1034] running on GCE node
I20260812 16:43:01.497773 16944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:01.499236 16944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:01.500447 16944 hybrid_clock.cc:648] HybridClock initialized: now 1786552981500348 us; error 91 us; skew 500 ppm
I20260812 16:43:01.500806 16944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:01 dist-test-slave-f7th krb5kdc[16930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552981, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:01.503851 16944 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:01.505314 16944 webserver.cc:460] Webserver started at http://127.31.61.190:45491/ using document root <none> and password file <none>
I20260812 16:43:01.505975 16944 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:01.506072 16944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:01.506325 16944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:01.509305 16944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "458188ae3bdf4086b62b16e95f89e954"
format_stamp: "Formatted at 2026-08-12 16:43:01 on dist-test-slave-f7th"
server_key: "950d480b4aee408e7b1b4c05f7160735"
server_key_iv: "9009cbc364f5b460e2f7046ad27c8994"
server_key_version: "encryptionkey@0"
I20260812 16:43:01.510015 16944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "458188ae3bdf4086b62b16e95f89e954"
format_stamp: "Formatted at 2026-08-12 16:43:01 on dist-test-slave-f7th"
server_key: "950d480b4aee408e7b1b4c05f7160735"
server_key_iv: "9009cbc364f5b460e2f7046ad27c8994"
server_key_version: "encryptionkey@0"
I20260812 16:43:01.515244 16944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:43:01.518816 16959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:01.520247 16944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:01.520412 16944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "458188ae3bdf4086b62b16e95f89e954"
format_stamp: "Formatted at 2026-08-12 16:43:01 on dist-test-slave-f7th"
server_key: "950d480b4aee408e7b1b4c05f7160735"
server_key_iv: "9009cbc364f5b460e2f7046ad27c8994"
server_key_version: "encryptionkey@0"
I20260812 16:43:01.520689 16944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:01.532722 16944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:01.536840 16944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:01.537118 16944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:01.555027 16944 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:34871
I20260812 16:43:01.555051 17010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:34871 every 8 connection(s)
I20260812 16:43:01.556478 16944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:01.558081 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 16944
I20260812 16:43:01.558302 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:01.559121 31990 external_mini_cluster.cc:1428] Setting key bf27622160c46aa45131662fdd3c2d1f
I20260812 16:43:01.561348 17011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 16:43:01 dist-test-slave-f7th krb5kdc[16930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:01.574786 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954: Bootstrap starting.
I20260812 16:43:01.578850 17011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:01.580160 17011 log.cc:824] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:01.583683 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954: No bootstrap required, opened a new log
I20260812 16:43:01.584000 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:01.561320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.31.61.190:34871)
0812 16:43:01.562130 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:01.562145 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:01.562194 (+ 49us) server_negotiation.cc:407] Connection header received
0812 16:43:01.564347 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:01.564385 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:01.564893 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:01.565347 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:01.566732 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:01.568762 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:01.570212 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:01.571230 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:01.571428 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:01.575787 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:01.575796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:01.575812 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:01.575856 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:01.579321 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:01.580338 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:01.580348 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:01.580358 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:01.580577 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:01.581036 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:01.581042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:01.581049 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:01.581764 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:01.581981 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:01.582476 (+ 495us) server_negotiation.cc:299] Negotiation successful
0812 16:43:01.582836 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":336,"threads_started":1}
I20260812 16:43:01.591006 17011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458188ae3bdf4086b62b16e95f89e954" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34871 } }
I20260812 16:43:01.591359 17011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:01.591457 17011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458188ae3bdf4086b62b16e95f89e954, State: Initialized, Role: FOLLOWER
I20260812 16:43:01.592051 17011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458188ae3bdf4086b62b16e95f89e954" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34871 } }
I20260812 16:43:01.592257 17011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:01.592406 17011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:01.592595 17011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:01.594805 17011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458188ae3bdf4086b62b16e95f89e954" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34871 } }
I20260812 16:43:01.595292 17011 leader_election.cc:302] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 458188ae3bdf4086b62b16e95f89e954; no voters:
I20260812 16:43:01.595919 17011 leader_election.cc:288] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:01.596057 17016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:01.596745 17016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [term 1 LEADER]: Becoming Leader. State: Replica: 458188ae3bdf4086b62b16e95f89e954, State: Running, Role: LEADER
I20260812 16:43:01.597383 17016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458188ae3bdf4086b62b16e95f89e954" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34871 } }
I20260812 16:43:01.598233 17011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:01.600041 17018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "458188ae3bdf4086b62b16e95f89e954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458188ae3bdf4086b62b16e95f89e954" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34871 } } }
I20260812 16:43:01.600577 17018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:01.601049 17017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458188ae3bdf4086b62b16e95f89e954. Latest consensus state: current_term: 1 leader_uuid: "458188ae3bdf4086b62b16e95f89e954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458188ae3bdf4086b62b16e95f89e954" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 34871 } } }
I20260812 16:43:01.601567 17017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:01.604190 17023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:01.609576 17023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:01.618959 17023 catalog_manager.cc:1347] Generated new cluster ID: 5ce5c43a71454deb861eb16b267e91fc
I20260812 16:43:01.619096 17023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:01.627650 17023 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:01.629590 17023 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:01.643155 17023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954: Generated new TSK 0
I20260812 16:43:01.644390 17023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:43:01.711367 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34871
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:02.048429 17037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:02.048715 17037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:02.048846 17037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:02.056267 17037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:02.056383 17037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:02.056528 17037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:43:02.066488 17037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:02.067868 17037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:02.069214 17037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:02.082247 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:02.082230 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:02.082731 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:02.083027 17037 server_base.cc:1034] running on GCE node
I20260812 16:43:02.083737 17037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:02.084702 17037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:02.085911 17037 hybrid_clock.cc:648] HybridClock initialized: now 1786552982085847 us; error 52 us; skew 500 ppm
I20260812 16:43:02.086211 17037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:02 dist-test-slave-f7th krb5kdc[16930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552982, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:02.089164 17037 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:43:02.090559 17037 webserver.cc:460] Webserver started at http://127.31.61.129:45001/ using document root <none> and password file <none>
I20260812 16:43:02.091236 17037 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:02.091352 17037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:02.091692 17037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:02.094538 17037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "353365d070064906b5581561ee9525af"
format_stamp: "Formatted at 2026-08-12 16:43:02 on dist-test-slave-f7th"
server_key: "f92b31671d0705babbcac727c441834a"
server_key_iv: "ce7b9bea5bc9a8a781fa1486281e394d"
server_key_version: "encryptionkey@0"
I20260812 16:43:02.095302 17037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "353365d070064906b5581561ee9525af"
format_stamp: "Formatted at 2026-08-12 16:43:02 on dist-test-slave-f7th"
server_key: "f92b31671d0705babbcac727c441834a"
server_key_iv: "ce7b9bea5bc9a8a781fa1486281e394d"
server_key_version: "encryptionkey@0"
I20260812 16:43:02.100837 17037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 16:43:02.104594 17052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:02.106046 17037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:43:02.106195 17037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "353365d070064906b5581561ee9525af"
format_stamp: "Formatted at 2026-08-12 16:43:02 on dist-test-slave-f7th"
server_key: "f92b31671d0705babbcac727c441834a"
server_key_iv: "ce7b9bea5bc9a8a781fa1486281e394d"
server_key_version: "encryptionkey@0"
I20260812 16:43:02.106380 17037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:02.119333 17037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:02.124092 17037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:02.124459 17037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:02.125501 17037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:02.126922 17037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:02.127003 17037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:02.127126 17037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:02.127182 17037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:02.160728 17037 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:38881
I20260812 16:43:02.160771 17165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:38881 every 8 connection(s)
I20260812 16:43:02.162009 17037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:43:02.168141 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 17037
I20260812 16:43:02.168332 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:43:02.169009 31990 external_mini_cluster.cc:1428] Setting key d3011b4d372d2f9091e0ed0dee6ba960
Aug 12 16:43:02 dist-test-slave-f7th krb5kdc[16930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552982, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:02.182458 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:02.164535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:33375 (local address 127.31.61.190:34871)
0812 16:43:02.164983 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:02.164988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:02.165875 (+ 887us) server_negotiation.cc:407] Connection header received
0812 16:43:02.167737 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:02.167745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:02.167835 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:02.167931 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:02.169502 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:02.170331 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:02.172058 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:02.172601 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:02.172747 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:02.176537 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:02.176543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:02.176547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:02.176587 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:02.179208 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:02.180074 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:02.180080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:02.180083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:02.180223 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:02.180636 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:02.180641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:02.180645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:02.180984 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:02.181150 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:02.181874 (+ 724us) server_negotiation.cc:299] Negotiation successful
0812 16:43:02.182118 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
I20260812 16:43:02.183269 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:02.165017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:34871 (local address 127.31.61.129:33375)
0812 16:43:02.165713 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:02.165758 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:02.167418 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:02.168180 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:02.168189 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:02.168734 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:02.169320 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:02.169339 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:02.170496 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:02.170500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:02.171906 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:02.171920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:02.172864 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:02.172872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:02.173112 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:02.173800 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:02.173827 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:02.176249 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:02.179381 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:02.179389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:02.179403 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:02.179924 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:02.180313 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:02.180316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:02.180318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:02.180533 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:02.181123 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:02.181130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:02.181496 (+ 366us) client_negotiation.cc:769] Sending connection context
0812 16:43:02.181723 (+ 227us) client_negotiation.cc:240] Negotiation successful
0812 16:43:02.182128 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":265,"threads_started":1}
I20260812 16:43:02.184906 17166 heartbeater.cc:347] Connected to a master server at 127.31.61.190:34871
I20260812 16:43:02.185258 17166 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:02.186156 17166 heartbeater.cc:511] Master 127.31.61.190:34871 requested a full tablet report, sending...
I20260812 16:43:02.188439 16976 ts_manager.cc:194] Registered new tserver with Master: 353365d070064906b5581561ee9525af (127.31.61.129:38881)
I20260812 16:43:02.189931 16976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:33375
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:43:02.227641 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34871
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:02.565299 17172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:02.565603 17172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:02.565723 17172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:02.573005 17172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:02.573137 17172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:02.573292 17172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:43:02.583190 17172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:02.584627 17172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:02.586015 17172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:02.599460 17177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:02.599586 17178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:02.599640 17180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:02.600244 17172 server_base.cc:1034] running on GCE node
I20260812 16:43:02.600891 17172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:02.601953 17172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:02.603196 17172 hybrid_clock.cc:648] HybridClock initialized: now 1786552982603139 us; error 48 us; skew 500 ppm
I20260812 16:43:02.603547 17172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:02 dist-test-slave-f7th krb5kdc[16930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552982, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:02.606809 17172 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:43:02.608438 17172 webserver.cc:460] Webserver started at http://127.31.61.130:43489/ using document root <none> and password file <none>
I20260812 16:43:02.609131 17172 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:02.609254 17172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:02.609586 17172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:02.612814 17172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c"
format_stamp: "Formatted at 2026-08-12 16:43:02 on dist-test-slave-f7th"
server_key: "581f2b5ca4d47aaaa2f2f08877d2219f"
server_key_iv: "42a11833ce7d7e316586fae09bdb30a4"
server_key_version: "encryptionkey@0"
I20260812 16:43:02.613649 17172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c"
format_stamp: "Formatted at 2026-08-12 16:43:02 on dist-test-slave-f7th"
server_key: "581f2b5ca4d47aaaa2f2f08877d2219f"
server_key_iv: "42a11833ce7d7e316586fae09bdb30a4"
server_key_version: "encryptionkey@0"
I20260812 16:43:02.619238 17172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 16:43:02.623159 17187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:02.624670 17172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 16:43:02.624822 17172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c"
format_stamp: "Formatted at 2026-08-12 16:43:02 on dist-test-slave-f7th"
server_key: "581f2b5ca4d47aaaa2f2f08877d2219f"
server_key_iv: "42a11833ce7d7e316586fae09bdb30a4"
server_key_version: "encryptionkey@0"
I20260812 16:43:02.625046 17172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:02.651736 17172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:02.656318 17172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:02.656591 17172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:02.657596 17172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:02.658974 17172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:02.659049 17172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:02.659139 17172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:02.659181 17172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:02.694000 17172 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:35547
I20260812 16:43:02.694089 17300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:35547 every 8 connection(s)
I20260812 16:43:02.695346 17172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:43:02.704869 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 17172
I20260812 16:43:02.705084 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:43:02.705646 31990 external_mini_cluster.cc:1428] Setting key 723501768efe508088d8daa25df80bb5
Aug 12 16:43:02 dist-test-slave-f7th krb5kdc[16930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552982, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:02.719961 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:02.698635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:50251 (local address 127.31.61.190:34871)
0812 16:43:02.699155 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:02.699160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:02.699940 (+ 780us) server_negotiation.cc:407] Connection header received
0812 16:43:02.701939 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:02.701945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:02.702035 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:02.702146 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:02.703997 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:02.705356 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:02.707918 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:02.708612 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:02.708762 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:02.712900 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:02.712908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:02.712913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:02.712960 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:02.716398 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:02.717302 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:02.717308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:02.717311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:02.717489 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:02.717980 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:02.717986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:02.717989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:02.718285 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:02.718402 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:02.719397 (+ 995us) server_negotiation.cc:299] Negotiation successful
0812 16:43:02.719646 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":249,"threads_started":1}
I20260812 16:43:02.721294 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:02.698831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:34871 (local address 127.31.61.130:50251)
0812 16:43:02.699733 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:02.699788 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:02.701629 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:02.702425 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:02.702434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:02.703001 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:02.703674 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:02.703697 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:02.705673 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:02.705679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:02.707703 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:02.707735 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:02.708920 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:02.708927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:02.709205 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:02.709942 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:02.709968 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:02.712577 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:02.716608 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:02.716617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:02.716633 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:02.717153 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:02.717625 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:02.717628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:02.717631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:02.717860 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:02.718418 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:02.718427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:02.718914 (+ 487us) client_negotiation.cc:769] Sending connection context
0812 16:43:02.719270 (+ 356us) client_negotiation.cc:240] Negotiation successful
0812 16:43:02.719865 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":323,"threads_started":1}
I20260812 16:43:02.722887 17301 heartbeater.cc:347] Connected to a master server at 127.31.61.190:34871
I20260812 16:43:02.723239 17301 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:02.724047 17301 heartbeater.cc:511] Master 127.31.61.190:34871 requested a full tablet report, sending...
I20260812 16:43:02.725668 16976 ts_manager.cc:194] Registered new tserver with Master: 54b981e0a9b24f18ab6b6a4a35fa6d9c (127.31.61.130:35547)
I20260812 16:43:02.726508 16976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:50251
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:43:02.771486 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34871
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:03.101032 17307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:03.101348 17307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:03.101487 17307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:03.108834 17307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:03.108965 17307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:03.109081 17307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:43:03.118973 17307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:34871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:03.120296 17307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:03.121644 17307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:03.135456 17312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:03.135432 17315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:03.135396 17313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:03.136174 17307 server_base.cc:1034] running on GCE node
I20260812 16:43:03.136824 17307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:03.137835 17307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:03.139067 17307 hybrid_clock.cc:648] HybridClock initialized: now 1786552983139014 us; error 45 us; skew 500 ppm
I20260812 16:43:03.139394 17307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:03 dist-test-slave-f7th krb5kdc[16930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552983, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:03.142717 17307 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:43:03.144333 17307 webserver.cc:460] Webserver started at http://127.31.61.131:39971/ using document root <none> and password file <none>
I20260812 16:43:03.145038 17307 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:03.145160 17307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:03.145494 17307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:03.148725 17307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1"
format_stamp: "Formatted at 2026-08-12 16:43:03 on dist-test-slave-f7th"
server_key: "5075152141530a8c884411d1442a8c36"
server_key_iv: "a1eca13e912a5abaa8c78b08a66b0915"
server_key_version: "encryptionkey@0"
I20260812 16:43:03.149531 17307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1"
format_stamp: "Formatted at 2026-08-12 16:43:03 on dist-test-slave-f7th"
server_key: "5075152141530a8c884411d1442a8c36"
server_key_iv: "a1eca13e912a5abaa8c78b08a66b0915"
server_key_version: "encryptionkey@0"
I20260812 16:43:03.155194 17307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 16:43:03.159138 17322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:03.160766 17307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:03.160953 17307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1"
format_stamp: "Formatted at 2026-08-12 16:43:03 on dist-test-slave-f7th"
server_key: "5075152141530a8c884411d1442a8c36"
server_key_iv: "a1eca13e912a5abaa8c78b08a66b0915"
server_key_version: "encryptionkey@0"
I20260812 16:43:03.161293 17307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:03.187206 17307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:03.191452 17307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:03.191772 17307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:03.192817 17307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:03.193704 17166 heartbeater.cc:503] Master 127.31.61.190:34871 was elected leader, sending a full tablet report...
I20260812 16:43:03.194317 17307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:03.194414 17307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:03.194541 17307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:03.194597 17307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:03.224659 17307 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:36007
I20260812 16:43:03.224745 17436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:36007 every 8 connection(s)
I20260812 16:43:03.225998 17307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:43:03.227385 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 17307
I20260812 16:43:03.227558 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:43:03.228159 31990 external_mini_cluster.cc:1428] Setting key 7a5f3f0b6b7920a6a26e3bfb6e00a61c
Aug 12 16:43:03 dist-test-slave-f7th krb5kdc[16930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552983, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:03.246882 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.229682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:42809 (local address 127.31.61.190:34871)
0812 16:43:03.230153 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.230159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.230510 (+ 351us) server_negotiation.cc:407] Connection header received
0812 16:43:03.232459 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.232466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.232560 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.232659 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:03.234161 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.234957 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.236525 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.236939 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.237046 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.240507 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:03.240512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:03.240516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:03.240554 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:03.243890 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.244714 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.244719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.244721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.244864 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.245294 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.245298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.245300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.245579 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:03.245702 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.246388 (+ 686us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.246599 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":253,"threads_started":1}
I20260812 16:43:03.248039 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.229172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:34871 (local address 127.31.61.131:42809)
0812 16:43:03.230342 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.230399 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:03.232201 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:03.232911 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:03.232919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:03.233439 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:03.233995 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.234013 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.235112 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.235116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.236383 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.236396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.237161 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.237169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.237404 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.238066 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:03.238091 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:03.240268 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:03.244062 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:03.244073 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:03.244093 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:03.244568 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:03.244971 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:03.244976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:03.244978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:03.245198 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:03.245706 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:03.245712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:03.246041 (+ 329us) client_negotiation.cc:769] Sending connection context
0812 16:43:03.246277 (+ 236us) client_negotiation.cc:240] Negotiation successful
0812 16:43:03.246718 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":310,"threads_started":1}
I20260812 16:43:03.249567 17437 heartbeater.cc:347] Connected to a master server at 127.31.61.190:34871
I20260812 16:43:03.249881 17437 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:03.250620 17437 heartbeater.cc:511] Master 127.31.61.190:34871 requested a full tablet report, sending...
I20260812 16:43:03.252200 16976 ts_manager.cc:194] Registered new tserver with Master: fc2e21bd0d5043bdb7a9e5507b2b5bb1 (127.31.61.131:36007)
I20260812 16:43:03.253052 16976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:42809
I20260812 16:43:03.257033 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:43:03.276525 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.262772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.31.61.190:34871)
0812 16:43:03.262976 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.262981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.263288 (+ 307us) server_negotiation.cc:407] Connection header received
0812 16:43:03.263459 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.263464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.263534 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.263655 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:03.265467 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.266385 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.267782 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.268254 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.268366 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.270820 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:03.270825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:03.270829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:03.270866 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:03.273770 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.274453 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.274457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.274464 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.274606 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.275107 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.275112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.275114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.275479 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:03.275721 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.276109 (+ 388us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.276292 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:43:03.280591 16976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:43:03.287550 16976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:43:03.330960 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.300604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.31.61.129:38881)
0812 16:43:03.322501 (+ 21897us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.322510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.322542 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:43:03.322645 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.322653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.322916 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.323098 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:03.323868 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.325739 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.328014 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.329785 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.329919 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.330176 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.330448 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.330628 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21699,"thread_start_us":21569,"threads_started":1}
I20260812 16:43:03.331167 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.300310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.0.0.1:41868)
0812 16:43:03.320918 (+ 20608us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.320934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:03.321032 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:03.323156 (+ 2124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:03.323165 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:03.323198 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:03.323626 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.323640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.325873 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.325878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.327848 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.327861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.330007 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.330013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.330202 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.330254 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 16:43:03.330374 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 16:43:03.330583 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19069,"mutex_wait_us":52,"thread_start_us":18876,"threads_started":1}
I20260812 16:43:03.331212 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.300657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:36007 (local address 127.0.0.1:36528)
0812 16:43:03.320694 (+ 20037us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.320758 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:03.320998 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:03.323455 (+ 2457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:03.323460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:03.323486 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:03.324160 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.324174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.326091 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.326096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.328450 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.328466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.330330 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.330337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.330527 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.330562 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:43:03.330630 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 16:43:03.330768 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19864,"mutex_wait_us":1,"thread_start_us":20855,"threads_started":1}
I20260812 16:43:03.332180 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.301383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.31.61.131:36007)
0812 16:43:03.322913 (+ 21530us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.322921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.322949 (+ 28us) server_negotiation.cc:407] Connection header received
0812 16:43:03.323050 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.323056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.323256 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.323437 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:03.324373 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.325955 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.328660 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.330089 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.331328 (+ 1239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.331578 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.331689 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.331857 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21371,"thread_start_us":21189,"threads_started":1}
I20260812 16:43:03.332737 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.301012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:35547 (local address 127.0.0.1:50964)
0812 16:43:03.321748 (+ 20736us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.321765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:03.321876 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:03.323324 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:03.323332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:03.323361 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:03.324131 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.324149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.326289 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.326294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.328536 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.328548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.332071 (+ 3523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.332076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.332191 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.332213 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 16:43:03.332280 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 16:43:03.332408 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20613,"mutex_wait_us":4,"thread_start_us":20143,"threads_started":1}
I20260812 16:43:03.333582 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.301918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.31.61.130:35547)
0812 16:43:03.322683 (+ 20765us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.322692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.322722 (+ 30us) server_negotiation.cc:407] Connection header received
0812 16:43:03.322822 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.322831 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.323067 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.323851 (+ 784us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:03.324379 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.326152 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.328735 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.331647 (+ 2912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.331784 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.331957 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.333086 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.333264 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20581,"thread_start_us":20435,"threads_started":1}
I20260812 16:43:03.335563 17101 tablet_service.cc:1467] Processing CreateTablet for tablet 0071a305be1440aba0bdccd7fed87382 (DEFAULT_TABLE table=test-table [id=c73a44a10f15497db1f982f59f6d1190]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:03.335829 17371 tablet_service.cc:1467] Processing CreateTablet for tablet 0071a305be1440aba0bdccd7fed87382 (DEFAULT_TABLE table=test-table [id=c73a44a10f15497db1f982f59f6d1190]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:03.336917 17101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0071a305be1440aba0bdccd7fed87382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:03.337277 17371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0071a305be1440aba0bdccd7fed87382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:03.337037 17235 tablet_service.cc:1467] Processing CreateTablet for tablet 0071a305be1440aba0bdccd7fed87382 (DEFAULT_TABLE table=test-table [id=c73a44a10f15497db1f982f59f6d1190]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:03.338353 17235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0071a305be1440aba0bdccd7fed87382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:03.348776 17456 tablet_bootstrap.cc:492] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: Bootstrap starting.
I20260812 16:43:03.352149 17456 tablet_bootstrap.cc:654] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:03.353085 17457 tablet_bootstrap.cc:492] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: Bootstrap starting.
I20260812 16:43:03.354482 17456 log.cc:824] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:03.354785 17458 tablet_bootstrap.cc:492] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: Bootstrap starting.
I20260812 16:43:03.356745 17457 tablet_bootstrap.cc:654] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:03.358058 17458 tablet_bootstrap.cc:654] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:03.358054 17457 log.cc:824] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:03.358645 17456 tablet_bootstrap.cc:492] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: No bootstrap required, opened a new log
I20260812 16:43:03.358969 17456 ts_tablet_manager.cc:1397] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260812 16:43:03.359215 17458 log.cc:824] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:03.361631 17457 tablet_bootstrap.cc:492] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: No bootstrap required, opened a new log
I20260812 16:43:03.361918 17457 ts_tablet_manager.cc:1397] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 16:43:03.362003 17458 tablet_bootstrap.cc:492] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: No bootstrap required, opened a new log
I20260812 16:43:03.362248 17458 ts_tablet_manager.cc:1397] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 16:43:03.369438 17458 raft_consensus.cc:348] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } }
I20260812 16:43:03.369853 17458 raft_consensus.cc:374] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:03.369946 17458 raft_consensus.cc:729] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b981e0a9b24f18ab6b6a4a35fa6d9c, State: Initialized, Role: FOLLOWER
I20260812 16:43:03.370575 17458 consensus_queue.cc:260] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } }
I20260812 16:43:03.370936 17456 raft_consensus.cc:348] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } }
I20260812 16:43:03.371389 17456 raft_consensus.cc:374] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:03.371822 17301 heartbeater.cc:503] Master 127.31.61.190:34871 was elected leader, sending a full tablet report...
I20260812 16:43:03.371824 17456 raft_consensus.cc:729] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353365d070064906b5581561ee9525af, State: Initialized, Role: FOLLOWER
I20260812 16:43:03.372478 17458 ts_tablet_manager.cc:1428] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260812 16:43:03.373281 17456 consensus_queue.cc:260] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } }
I20260812 16:43:03.373932 17457 raft_consensus.cc:348] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } }
I20260812 16:43:03.374361 17457 raft_consensus.cc:374] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:03.374462 17457 raft_consensus.cc:729] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2e21bd0d5043bdb7a9e5507b2b5bb1, State: Initialized, Role: FOLLOWER
I20260812 16:43:03.375149 17456 ts_tablet_manager.cc:1428] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260812 16:43:03.375137 17457 consensus_queue.cc:260] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } }
I20260812 16:43:03.377504 17437 heartbeater.cc:503] Master 127.31.61.190:34871 was elected leader, sending a full tablet report...
I20260812 16:43:03.378144 17457 ts_tablet_manager.cc:1428] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
W20260812 16:43:03.414492 17167 tablet.cc:2369] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:03.438482 17464 raft_consensus.cc:482] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:43:03.438827 17464 raft_consensus.cc:504] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } }
I20260812 16:43:03.440601 17464 leader_election.cc:288] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 353365d070064906b5581561ee9525af (127.31.61.129:38881), 54b981e0a9b24f18ab6b6a4a35fa6d9c (127.31.61.130:35547)
W20260812 16:43:03.447719 17302 tablet.cc:2369] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:03.448302 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.441337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39855 (local address 127.31.61.129:38881)
0812 16:43:03.441574 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.441579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.441598 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:43:03.441676 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.441681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.441758 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.441875 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:03.442923 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.443854 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.446065 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.447149 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.447390 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.447481 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.447860 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.448012 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:43:03.448720 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.440993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.31.61.131:39855)
0812 16:43:03.441293 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.441315 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:03.441465 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:03.442040 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:03.442047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:03.442084 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:03.442690 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.442706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.444033 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.444038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.445877 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.445893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.447425 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.447431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.447678 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.447715 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 16:43:03.447813 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 16:43:03.448237 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260812 16:43:03.449265 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.441353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:57567 (local address 127.31.61.130:35547)
0812 16:43:03.441562 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.441569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.442897 (+ 1328us) server_negotiation.cc:407] Connection header received
0812 16:43:03.443006 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.443011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.443087 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.443186 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:03.443775 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.444543 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.446799 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.448100 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.448257 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.448338 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.448851 (+ 513us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.448993 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:43:03.449383 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.441119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:35547 (local address 127.31.61.131:57567)
0812 16:43:03.442774 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.442792 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:03.442908 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:03.443270 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:03.443274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:03.443297 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:03.443614 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.443626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.444706 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.444711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.446580 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:03.446595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.448490 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.448497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:03.448678 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.448709 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 16:43:03.448803 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 16:43:03.448951 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"mutex_wait_us":465,"thread_start_us":413,"threads_started":1}
I20260812 16:43:03.450116 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0071a305be1440aba0bdccd7fed87382" candidate_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353365d070064906b5581561ee9525af" is_pre_election: true
I20260812 16:43:03.450441 17255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0071a305be1440aba0bdccd7fed87382" candidate_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" is_pre_election: true
I20260812 16:43:03.450793 17121 raft_consensus.cc:2454] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc2e21bd0d5043bdb7a9e5507b2b5bb1 in term 0.
I20260812 16:43:03.450981 17255 raft_consensus.cc:2454] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc2e21bd0d5043bdb7a9e5507b2b5bb1 in term 0.
I20260812 16:43:03.451565 17324 leader_election.cc:302] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353365d070064906b5581561ee9525af, fc2e21bd0d5043bdb7a9e5507b2b5bb1; no voters:
I20260812 16:43:03.452028 17464 raft_consensus.cc:2781] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:43:03.452171 17464 raft_consensus.cc:482] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:43:03.452278 17464 raft_consensus.cc:3037] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:03.454666 17464 raft_consensus.cc:504] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } }
I20260812 16:43:03.455375 17464 leader_election.cc:288] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [CANDIDATE]: Term 1 election: Requested vote from peers 353365d070064906b5581561ee9525af (127.31.61.129:38881), 54b981e0a9b24f18ab6b6a4a35fa6d9c (127.31.61.130:35547)
I20260812 16:43:03.455847 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0071a305be1440aba0bdccd7fed87382" candidate_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353365d070064906b5581561ee9525af"
I20260812 16:43:03.455976 17255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0071a305be1440aba0bdccd7fed87382" candidate_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c"
I20260812 16:43:03.456045 17121 raft_consensus.cc:3037] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:03.456195 17255 raft_consensus.cc:3037] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:03.458379 17255 raft_consensus.cc:2454] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc2e21bd0d5043bdb7a9e5507b2b5bb1 in term 1.
I20260812 16:43:03.458387 17121 raft_consensus.cc:2454] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc2e21bd0d5043bdb7a9e5507b2b5bb1 in term 1.
I20260812 16:43:03.458837 17325 leader_election.cc:302] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b981e0a9b24f18ab6b6a4a35fa6d9c, fc2e21bd0d5043bdb7a9e5507b2b5bb1; no voters:
I20260812 16:43:03.459184 17464 raft_consensus.cc:2781] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:03.459681 17464 raft_consensus.cc:686] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 1 LEADER]: Becoming Leader. State: Replica: fc2e21bd0d5043bdb7a9e5507b2b5bb1, State: Running, Role: LEADER
I20260812 16:43:03.460224 17464 consensus_queue.cc:237] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } }
I20260812 16:43:03.464752 16976 catalog_manager.cc:5495] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc2e21bd0d5043bdb7a9e5507b2b5bb1 (127.31.61.131). New cstate: current_term: 1 leader_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc2e21bd0d5043bdb7a9e5507b2b5bb1" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 36007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 } health_report { overall_health: UNKNOWN } } }
W20260812 16:43:03.477525 17438 tablet.cc:2369] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:03.489758 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.484249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.31.61.131:36007)
0812 16:43:03.484441 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.484447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.484528 (+ 81us) server_negotiation.cc:407] Connection header received
0812 16:43:03.484760 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.484764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.484842 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.484967 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:43:03.485367 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.486073 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.487665 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.488065 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.488176 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.488669 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:43:03.489175 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:43:03.489311 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.489413 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.489497 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 16:43:03.497740 17255 raft_consensus.cc:1261] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [term 1 FOLLOWER]: Refusing update from remote peer fc2e21bd0d5043bdb7a9e5507b2b5bb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:03.497740 17121 raft_consensus.cc:1261] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [term 1 FOLLOWER]: Refusing update from remote peer fc2e21bd0d5043bdb7a9e5507b2b5bb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:03.498642 17466 consensus_queue.cc:1035] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "353365d070064906b5581561ee9525af" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:03.498968 17464 consensus_queue.cc:1035] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54b981e0a9b24f18ab6b6a4a35fa6d9c" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:03.509758 17469 mvcc.cc:205] Tried to move back new op lower bound from 7317721020397400064 to 7317721020256972800. Current Snapshot: MvccSnapshot[applied={T|T < 7317721020397400064}]
I20260812 16:43:03.523080 16976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:03.523346 16976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:03.527316 16976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 458188ae3bdf4086b62b16e95f89e954: Sending DeleteTablet for 3 replicas of tablet 0071a305be1440aba0bdccd7fed87382
I20260812 16:43:03.528633 17371 tablet_service.cc:1514] Processing DeleteTablet for tablet 0071a305be1440aba0bdccd7fed87382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:03 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:36528
I20260812 16:43:03.528649 17101 tablet_service.cc:1514] Processing DeleteTablet for tablet 0071a305be1440aba0bdccd7fed87382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:03 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:41868
I20260812 16:43:03.529835 17235 tablet_service.cc:1514] Processing DeleteTablet for tablet 0071a305be1440aba0bdccd7fed87382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:03 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:50964
I20260812 16:43:03.531829 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 17037
I20260812 16:43:03.539932 17479 tablet_replica.cc:330] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: stopping tablet replica
I20260812 16:43:03.540591 17479 raft_consensus.cc:2229] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:03.541275 17479 raft_consensus.cc:2258] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:03.541937 17480 tablet_replica.cc:330] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: stopping tablet replica
I20260812 16:43:03.542832 17481 tablet_replica.cc:330] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: stopping tablet replica
I20260812 16:43:03.543272 17481 raft_consensus.cc:2229] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:03.543838 17481 raft_consensus.cc:2258] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:03.543848 17480 raft_consensus.cc:2229] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:43:03.545408 17480 raft_consensus.cc:2258] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:03.545665 17481 ts_tablet_manager.cc:1905] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:03.546216 17479 ts_tablet_manager.cc:1905] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 16:43:03 dist-test-slave-f7th krb5kdc[16930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:43:03.553267 17481 ts_tablet_manager.cc:1918] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:03.553452 17481 log.cc:1196] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/0071a305be1440aba0bdccd7fed87382
I20260812 16:43:03.553341 17479 ts_tablet_manager.cc:1918] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:03.553746 17479 log.cc:1196] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/0071a305be1440aba0bdccd7fed87382
I20260812 16:43:03.554003 17481 ts_tablet_manager.cc:1939] T 0071a305be1440aba0bdccd7fed87382 P 54b981e0a9b24f18ab6b6a4a35fa6d9c: Deleting consensus metadata
I20260812 16:43:03.556684 17480 ts_tablet_manager.cc:1905] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:03.558279 16962 catalog_manager.cc:4841] TS 54b981e0a9b24f18ab6b6a4a35fa6d9c (127.31.61.130:35547): tablet 0071a305be1440aba0bdccd7fed87382 (table test-table [id=c73a44a10f15497db1f982f59f6d1190]) successfully deleted
I20260812 16:43:03.561596 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.533366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.31.61.129:38881)
0812 16:43:03.533561 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.533568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.542168 (+ 8600us) server_negotiation.cc:407] Connection header received
0812 16:43:03.542375 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.542381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.542466 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.542591 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:03.544904 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.545939 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.547909 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.548563 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.548657 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.554094 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:03.554110 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:03.554129 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:03.554181 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:03.557816 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.558883 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.558892 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.558897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.559065 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.559544 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.559550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.559554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.560056 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:03.560355 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.560969 (+ 614us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.561226 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:43:03.563061 17161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:03.564059 17480 ts_tablet_manager.cc:1918] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:03.564254 17480 log.cc:1196] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/0071a305be1440aba0bdccd7fed87382
I20260812 16:43:03.564967 17480 ts_tablet_manager.cc:1939] T 0071a305be1440aba0bdccd7fed87382 P fc2e21bd0d5043bdb7a9e5507b2b5bb1: Deleting consensus metadata
I20260812 16:43:03.567090 16963 catalog_manager.cc:4841] TS fc2e21bd0d5043bdb7a9e5507b2b5bb1 (127.31.61.131:36007): tablet 0071a305be1440aba0bdccd7fed87382 (table test-table [id=c73a44a10f15497db1f982f59f6d1190]) successfully deleted
I20260812 16:43:03.683933 17479 ts_tablet_manager.cc:1939] T 0071a305be1440aba0bdccd7fed87382 P 353365d070064906b5581561ee9525af: Deleting consensus metadata
I20260812 16:43:03.684669 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 17037
I20260812 16:43:03.706558 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 17172
W20260812 16:43:03.707070 16961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.61.129:38881 (error 108)
I20260812 16:43:03.716128 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.715498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.0.0.1:41876)
0812 16:43:03.715789 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.715947 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 16:43:03.716420 16961 catalog_manager.cc:4584] TS 353365d070064906b5581561ee9525af (127.31.61.129:38881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0071a305be1440aba0bdccd7fed87382: Network error: Client connection negotiation failed: client connection to 127.31.61.129:38881: connect: Connection refused (error 111)
Aug 12 16:43:03 dist-test-slave-f7th krb5kdc[16930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:43:03.726585 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.707726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.31.61.130:35547)
0812 16:43:03.707901 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.707907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.708316 (+ 409us) server_negotiation.cc:407] Connection header received
0812 16:43:03.708388 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.708393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.708487 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.708577 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:03.710353 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.711550 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.713625 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.714247 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.714369 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.719257 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:03.719275 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:03.719294 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:03.719348 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:03.723232 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.724138 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.724149 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.724153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.724316 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.724752 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.724757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.724760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.725313 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:03.725601 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.725946 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.726197 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 16:43:03.727342 17295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:03.770457 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.769873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.0.0.1:41882)
0812 16:43:03.770125 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.770247 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 16:43:03.771698 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.771255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.0.0.1:41898)
0812 16:43:03.771416 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.771525 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 16:43:03.819870 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 17172
I20260812 16:43:03.832605 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.832146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.0.0.1:41910)
0812 16:43:03.832349 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.832436 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 16:43:03.833771 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.833409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.0.0.1:41914)
0812 16:43:03.833565 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.833642 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 16:43:03.842782 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 17307
Aug 12 16:43:03 dist-test-slave-f7th krb5kdc[16930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:43:03.859979 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.843805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.31.61.131:36007)
0812 16:43:03.843973 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:03.843978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:03.844132 (+ 154us) server_negotiation.cc:407] Connection header received
0812 16:43:03.844304 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:03.844309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:03.844395 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:03.844481 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:03.845840 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.846694 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.848092 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:03.848576 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:03.848703 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:03.853521 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:03.853536 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:03.853553 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:03.853607 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:03.856929 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.857683 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.857690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.857692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.857870 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:03.858258 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:03.858261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:03.858263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:03.858656 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:03.858968 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:03.859309 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 16:43:03.859548 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 16:43:03.860615 17432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:03.914247 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.913779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.0.0.1:41928)
0812 16:43:03.913981 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.914071 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 16:43:03.915370 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:03.914981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.0.0.1:41944)
0812 16:43:03.915186 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:03.915258 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 16:43:03.984309 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 17307
I20260812 16:43:04.003146 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 16944
I20260812 16:43:04.004528 17006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:04.116555 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:04.115753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38881 (local address 127.0.0.1:41952)
0812 16:43:04.115953 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:04.116056 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 16:43:04.116664 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 16944
2026-08-12T16:43:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4312 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T16:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:04Z Disabled control of system clock
I20260812 16:43:04.194375 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36199
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:43:04.538398 17491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:04.538751 17491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:04.538866 17491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:04.546752 17491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:04.546900 17491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:04.546937 17491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:04.546988 17491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:04.547032 17491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:04.556993 17491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:04.558503 17491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:04.560405 17491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:04.572232 17498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:04.572495 17491 server_base.cc:1034] running on GCE node
W20260812 16:43:04.572638 17497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:04.572427 17500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:04.573630 17491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:04.575214 17491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:04.576479 17491 hybrid_clock.cc:648] HybridClock initialized: now 1786552984576396 us; error 46 us; skew 500 ppm
I20260812 16:43:04.576879 17491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:04.583331 17491 webserver.cc:460] Webserver started at http://127.31.61.190:43305/ using document root <none> and password file <none>
I20260812 16:43:04.584160 17491 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:04.584277 17491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:04.584606 17491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:04.587659 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "ca879d154f4b40a588d204f46c767c42"
format_stamp: "Formatted at 2026-08-12 16:43:04 on dist-test-slave-f7th"
server_key: "c5a4ba2bcbbf0628935764432c66107c"
server_key_iv: "fa6119c63faec1460615885a91a28696"
server_key_version: "encryptionkey@0"
I20260812 16:43:04.588501 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "ca879d154f4b40a588d204f46c767c42"
format_stamp: "Formatted at 2026-08-12 16:43:04 on dist-test-slave-f7th"
server_key: "c5a4ba2bcbbf0628935764432c66107c"
server_key_iv: "fa6119c63faec1460615885a91a28696"
server_key_version: "encryptionkey@0"
I20260812 16:43:04.593927 17491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:43:04.597693 17506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:04.599237 17491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:43:04.599462 17491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "ca879d154f4b40a588d204f46c767c42"
format_stamp: "Formatted at 2026-08-12 16:43:04 on dist-test-slave-f7th"
server_key: "c5a4ba2bcbbf0628935764432c66107c"
server_key_iv: "fa6119c63faec1460615885a91a28696"
server_key_version: "encryptionkey@0"
I20260812 16:43:04.599720 17491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:04.620157 17491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:04.621047 17491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:04.621335 17491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:04.639843 17491 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36199
I20260812 16:43:04.639863 17557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36199 every 8 connection(s)
I20260812 16:43:04.641291 17491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:04.645043 17558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:04.650707 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 17491
I20260812 16:43:04.650895 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:04.651449 31990 external_mini_cluster.cc:1428] Setting key ef8e9001e1952c02b97d4e69064c3a56
I20260812 16:43:04.656576 17558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42: Bootstrap starting.
I20260812 16:43:04.659747 17558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:04.660759 17558 log.cc:824] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:04.663993 17558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42: No bootstrap required, opened a new log
I20260812 16:43:04.664992 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:04.653583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.31.61.190:36199)
0812 16:43:04.654629 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:04.654641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:04.654684 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:43:04.656486 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:04.656518 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:04.657008 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:04.657380 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:04.657928 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:04.659210 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:04.660625 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:04.661411 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:04.661605 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:04.662699 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:04.662707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:04.662721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:04.663260 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:04.663345 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:04.663368 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:04.663457 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:04.663732 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 16:43:04.664035 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":363,"threads_started":1}
I20260812 16:43:04.671314 17558 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca879d154f4b40a588d204f46c767c42" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36199 } }
I20260812 16:43:04.671691 17558 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:04.671787 17558 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca879d154f4b40a588d204f46c767c42, State: Initialized, Role: FOLLOWER
I20260812 16:43:04.672346 17558 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca879d154f4b40a588d204f46c767c42" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36199 } }
I20260812 16:43:04.672557 17558 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:04.672690 17558 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:04.672848 17558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:04.674966 17558 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca879d154f4b40a588d204f46c767c42" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36199 } }
I20260812 16:43:04.675459 17558 leader_election.cc:302] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca879d154f4b40a588d204f46c767c42; no voters:
I20260812 16:43:04.676122 17558 leader_election.cc:288] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:04.676322 17563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:04.677268 17563 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [term 1 LEADER]: Becoming Leader. State: Replica: ca879d154f4b40a588d204f46c767c42, State: Running, Role: LEADER
I20260812 16:43:04.677814 17563 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca879d154f4b40a588d204f46c767c42" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36199 } }
I20260812 16:43:04.678707 17558 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:04.680400 17565 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca879d154f4b40a588d204f46c767c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca879d154f4b40a588d204f46c767c42" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36199 } } }
I20260812 16:43:04.680817 17565 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:04.682574 17564 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca879d154f4b40a588d204f46c767c42. Latest consensus state: current_term: 1 leader_uuid: "ca879d154f4b40a588d204f46c767c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca879d154f4b40a588d204f46c767c42" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36199 } } }
I20260812 16:43:04.682895 17564 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:04.683043 17570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:04.687292 17570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:04.695470 17570 catalog_manager.cc:1347] Generated new cluster ID: 68f20310f84c4594b3d9fe4f91fa9c16
I20260812 16:43:04.695609 17570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:04.714778 17570 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:04.716634 17570 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:04.733450 17570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42: Generated new TSK 0
I20260812 16:43:04.734539 17570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:04.746863 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36199
--builtin_ntp_servers=127.31.61.148:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:43:05.085026 17582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:05.085302 17582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:05.085420 17582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:05.092767 17582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:05.092864 17582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:05.092995 17582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:43:05.102207 17582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:05.103454 17582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:05.105273 17582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:05.117704 17587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:05.117748 17588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:05.117913 17590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:05.118541 17582 server_base.cc:1034] running on GCE node
I20260812 16:43:05.119167 17582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:05.120193 17582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:05.121425 17582 hybrid_clock.cc:648] HybridClock initialized: now 1786552985121356 us; error 53 us; skew 500 ppm
I20260812 16:43:05.121784 17582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:05.124202 17582 webserver.cc:460] Webserver started at http://127.31.61.129:38601/ using document root <none> and password file <none>
I20260812 16:43:05.124913 17582 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:05.125021 17582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:05.125406 17582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:05.128450 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "2b1264ef4de44f309b16e16ef0dff1a7"
format_stamp: "Formatted at 2026-08-12 16:43:05 on dist-test-slave-f7th"
server_key: "d9c10c1d041342d7e2b011ec17ba8e14"
server_key_iv: "aac5fcd1607670dfa157e3a6dcbd51a5"
server_key_version: "encryptionkey@0"
I20260812 16:43:05.129235 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "2b1264ef4de44f309b16e16ef0dff1a7"
format_stamp: "Formatted at 2026-08-12 16:43:05 on dist-test-slave-f7th"
server_key: "d9c10c1d041342d7e2b011ec17ba8e14"
server_key_iv: "aac5fcd1607670dfa157e3a6dcbd51a5"
server_key_version: "encryptionkey@0"
I20260812 16:43:05.134815 17582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:43:05.138701 17596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:05.140244 17582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:05.140408 17582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "2b1264ef4de44f309b16e16ef0dff1a7"
format_stamp: "Formatted at 2026-08-12 16:43:05 on dist-test-slave-f7th"
server_key: "d9c10c1d041342d7e2b011ec17ba8e14"
server_key_iv: "aac5fcd1607670dfa157e3a6dcbd51a5"
server_key_version: "encryptionkey@0"
I20260812 16:43:05.140631 17582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:05.190922 17582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:05.191895 17582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:05.192178 17582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:05.193207 17582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:05.194696 17582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:05.194790 17582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:05.194870 17582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:05.194922 17582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:05.227253 17582 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:44103
I20260812 16:43:05.227309 17709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:44103 every 8 connection(s)
I20260812 16:43:05.228585 17582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:43:05.232833 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 17582
I20260812 16:43:05.233034 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:43:05.233502 31990 external_mini_cluster.cc:1428] Setting key f3eb26372e3968fdc89a3bc63d90a43e
I20260812 16:43:05.236330 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36199
--builtin_ntp_servers=127.31.61.148:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:43:05.242421 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:05.231343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:45801 (local address 127.31.61.190:36199)
0812 16:43:05.231842 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:05.231847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:05.233293 (+ 1446us) server_negotiation.cc:407] Connection header received
0812 16:43:05.235104 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:05.235110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:05.235190 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:05.235283 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:05.236571 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:05.237276 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:05.239403 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:05.239924 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:05.240044 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:05.241580 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:05.241586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:05.241589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:05.241608 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:05.241657 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:05.241671 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:05.241749 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:05.242028 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 16:43:05.242120 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":275,"threads_started":1}
I20260812 16:43:05.243669 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:05.231886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36199 (local address 127.31.61.129:45801)
0812 16:43:05.232903 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:05.233171 (+ 268us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:05.234842 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:05.235649 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:05.235660 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:05.235754 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:05.236356 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:05.236374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:05.237450 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:05.237457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:05.239217 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:05.239233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:05.240153 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:05.240159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:05.240428 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:05.241194 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:43:05.241198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:05.241250 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:43:05.241350 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:05.241765 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:05.241777 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:05.241784 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 16:43:05.241972 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 16:43:05.242389 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":287,"threads_started":1}
I20260812 16:43:05.245280 17710 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36199
I20260812 16:43:05.245595 17710 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:05.246564 17710 heartbeater.cc:511] Master 127.31.61.190:36199 requested a full tablet report, sending...
I20260812 16:43:05.248744 17523 ts_manager.cc:194] Registered new tserver with Master: 2b1264ef4de44f309b16e16ef0dff1a7 (127.31.61.129:44103)
I20260812 16:43:05.250231 17523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.129:45801
W20260812 16:43:05.574429 17714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:05.574709 17714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:05.574824 17714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:05.582208 17714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:05.582321 17714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:05.582466 17714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:43:05.591913 17714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:05.593235 17714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:05.595029 17714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:05.608176 17719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:05.608238 17720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:05.608377 17722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:05.609112 17714 server_base.cc:1034] running on GCE node
I20260812 16:43:05.609710 17714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:05.610651 17714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:05.611893 17714 hybrid_clock.cc:648] HybridClock initialized: now 1786552985611828 us; error 46 us; skew 500 ppm
I20260812 16:43:05.612223 17714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:05.614544 17714 webserver.cc:460] Webserver started at http://127.31.61.130:33059/ using document root <none> and password file <none>
I20260812 16:43:05.615273 17714 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:05.615375 17714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:05.615725 17714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:05.618825 17714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "5e7cdd2f29f2471781f230a3188c4438"
format_stamp: "Formatted at 2026-08-12 16:43:05 on dist-test-slave-f7th"
server_key: "02ba390252ab7c922426a887f8feac26"
server_key_iv: "ed5d0c7c96cae79e0ba5f3105aaf3fdf"
server_key_version: "encryptionkey@0"
I20260812 16:43:05.619676 17714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "5e7cdd2f29f2471781f230a3188c4438"
format_stamp: "Formatted at 2026-08-12 16:43:05 on dist-test-slave-f7th"
server_key: "02ba390252ab7c922426a887f8feac26"
server_key_iv: "ed5d0c7c96cae79e0ba5f3105aaf3fdf"
server_key_version: "encryptionkey@0"
I20260812 16:43:05.625149 17714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 16:43:05.629078 17728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:05.630604 17714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:43:05.630776 17714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "5e7cdd2f29f2471781f230a3188c4438"
format_stamp: "Formatted at 2026-08-12 16:43:05 on dist-test-slave-f7th"
server_key: "02ba390252ab7c922426a887f8feac26"
server_key_iv: "ed5d0c7c96cae79e0ba5f3105aaf3fdf"
server_key_version: "encryptionkey@0"
I20260812 16:43:05.631003 17714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:05.648663 17714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:05.649535 17714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:05.649835 17714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:05.650820 17714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:05.652449 17714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:05.652555 17714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:05.652663 17714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:05.652732 17714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:05.685413 17714 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:42611
I20260812 16:43:05.685492 17841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:42611 every 8 connection(s)
I20260812 16:43:05.686753 17714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:43:05.696879 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 17714
I20260812 16:43:05.697190 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:43:05.697805 31990 external_mini_cluster.cc:1428] Setting key 28901328788156b80e0c82add2d4860c
I20260812 16:43:05.700076 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:05.689415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:38799 (local address 127.31.61.190:36199)
0812 16:43:05.689598 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:05.689604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:05.690955 (+ 1351us) server_negotiation.cc:407] Connection header received
0812 16:43:05.692846 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:05.692850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:05.692923 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:05.693014 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:05.694078 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:05.694820 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:05.696670 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:05.697338 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:05.697487 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:05.699227 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:05.699232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:05.699235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:05.699254 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:05.699288 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:05.699300 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:05.699375 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:05.699651 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 16:43:05.699756 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 16:43:05.700722 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36199
--builtin_ntp_servers=127.31.61.148:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:43:05.701216 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:05.689908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36199 (local address 127.31.61.130:38799)
0812 16:43:05.690799 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:05.690834 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:05.692567 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:05.693211 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:05.693219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:05.693304 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:05.693885 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:05.693937 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:05.694991 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:05.694996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:05.696479 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:05.696496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:05.697626 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:05.697631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:05.697847 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:05.698837 (+ 990us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:43:05.698842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:05.698901 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:43:05.699017 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:05.699418 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:05.699431 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:05.699437 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 16:43:05.699625 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 16:43:05.700059 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":263,"threads_started":1}
I20260812 16:43:05.702690 17842 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36199
I20260812 16:43:05.703009 17842 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:05.703900 17842 heartbeater.cc:511] Master 127.31.61.190:36199 requested a full tablet report, sending...
I20260812 16:43:05.705487 17523 ts_manager.cc:194] Registered new tserver with Master: 5e7cdd2f29f2471781f230a3188c4438 (127.31.61.130:42611)
I20260812 16:43:05.706524 17523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.130:38799
W20260812 16:43:06.038923 17845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:06.039244 17845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:06.039357 17845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:06.046993 17845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:06.047099 17845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:06.047233 17845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:43:06.056784 17845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:36199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:06.058109 17845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:06.059944 17845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:06.073220 17850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:06.073325 17851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:06.073251 17853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:06.074445 17845 server_base.cc:1034] running on GCE node
I20260812 16:43:06.075098 17845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:06.076133 17845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:06.077378 17845 hybrid_clock.cc:648] HybridClock initialized: now 1786552986077302 us; error 62 us; skew 500 ppm
I20260812 16:43:06.077718 17845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:06.080278 17845 webserver.cc:460] Webserver started at http://127.31.61.131:38493/ using document root <none> and password file <none>
I20260812 16:43:06.080974 17845 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:06.081073 17845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:06.081403 17845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:06.084374 17845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "811e176889bd47bb83a074ae6f3756d2"
format_stamp: "Formatted at 2026-08-12 16:43:06 on dist-test-slave-f7th"
server_key: "af86ec61d0cd68b660346bcb5781c91c"
server_key_iv: "568eaf867d6f609a1b9f5f11172af93a"
server_key_version: "encryptionkey@0"
I20260812 16:43:06.085152 17845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "811e176889bd47bb83a074ae6f3756d2"
format_stamp: "Formatted at 2026-08-12 16:43:06 on dist-test-slave-f7th"
server_key: "af86ec61d0cd68b660346bcb5781c91c"
server_key_iv: "568eaf867d6f609a1b9f5f11172af93a"
server_key_version: "encryptionkey@0"
I20260812 16:43:06.090332 17845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 16:43:06.093911 17859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:06.095335 17845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:43:06.095504 17845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "811e176889bd47bb83a074ae6f3756d2"
format_stamp: "Formatted at 2026-08-12 16:43:06 on dist-test-slave-f7th"
server_key: "af86ec61d0cd68b660346bcb5781c91c"
server_key_iv: "568eaf867d6f609a1b9f5f11172af93a"
server_key_version: "encryptionkey@0"
I20260812 16:43:06.095788 17845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:06.111831 17845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:06.112690 17845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:06.112963 17845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:06.113910 17845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:06.115517 17845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:06.115635 17845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:06.115743 17845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:06.115803 17845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:06.148468 17845 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:43725
I20260812 16:43:06.148499 17972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:43725 every 8 connection(s)
I20260812 16:43:06.149796 17845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:43:06.159073 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 17845
I20260812 16:43:06.159283 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:43:06.159976 31990 external_mini_cluster.cc:1428] Setting key 85acc64bfae7429c4a1e41e17dabe336
I20260812 16:43:06.164144 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.152873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:45935 (local address 127.31.61.190:36199)
0812 16:43:06.153124 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.153131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.154616 (+ 1485us) server_negotiation.cc:407] Connection header received
0812 16:43:06.156494 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.156500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.156576 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.156672 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:06.158345 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.159117 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.161388 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.161947 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.162048 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.163451 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:06.163456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:06.163460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:06.163477 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:06.163512 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:06.163523 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:06.163630 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.163818 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.163898 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 16:43:06.165179 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.153428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36199 (local address 127.31.61.131:45935)
0812 16:43:06.154404 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:06.154475 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:06.156198 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:06.157302 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:06.157317 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:06.157475 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:06.158130 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.158152 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.159317 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.159324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.161202 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.161220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.162167 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.162173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.162392 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.163148 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:43:06.163150 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:06.163194 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:43:06.163279 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:06.163639 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:06.163649 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:06.163656 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 16:43:06.163805 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 16:43:06.164129 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":313,"threads_started":1}
I20260812 16:43:06.166522 17973 heartbeater.cc:347] Connected to a master server at 127.31.61.190:36199
I20260812 16:43:06.166792 17973 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:06.167528 17973 heartbeater.cc:511] Master 127.31.61.190:36199 requested a full tablet report, sending...
I20260812 16:43:06.169040 17523 ts_manager.cc:194] Registered new tserver with Master: 811e176889bd47bb83a074ae6f3756d2 (127.31.61.131:43725)
I20260812 16:43:06.169945 17523 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.131:45935
I20260812 16:43:06.175273 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:43:06.185343 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.179385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.31.61.190:36199)
0812 16:43:06.179626 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.179632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.179936 (+ 304us) server_negotiation.cc:407] Connection header received
0812 16:43:06.180116 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.180120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.180197 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.180277 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:06.180688 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.181406 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.182765 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.183207 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.183301 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.184569 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:06.184573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:06.184576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:06.184595 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:06.184631 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:06.184643 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:06.184721 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.184932 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.185023 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 16:43:06.189240 17523 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:43:06.196265 17523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:43:06.218812 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.209004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:42611 (local address 127.0.0.1:57744)
0812 16:43:06.210640 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:06.210657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:06.210763 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:06.212653 (+ 1890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:06.212658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:06.212688 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:06.213034 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.213045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.214745 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.214750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.216625 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.216636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.218137 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.218143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.218316 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.218351 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:43:06.218413 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 16:43:06.218520 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1517,"mutex_wait_us":3,"thread_start_us":2053,"threads_started":1}
I20260812 16:43:06.218874 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.208433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:43725 (local address 127.0.0.1:39426)
0812 16:43:06.209504 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:06.209548 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:06.209745 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:06.211003 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:06.211011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:06.211055 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:06.211526 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.211538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.213553 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.213559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.216014 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.216030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.217907 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.217914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.218138 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.218199 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 16:43:06.218303 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 16:43:06.218497 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":21,"thread_start_us":552,"threads_started":1}
I20260812 16:43:06.220080 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.208729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.31.61.131:43725)
0812 16:43:06.210381 (+ 1652us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.210389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.210423 (+ 34us) server_negotiation.cc:407] Connection header received
0812 16:43:06.210534 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.210541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.210781 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.211642 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:06.211914 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.213384 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.216233 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.217684 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.219152 (+ 1468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.219434 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.219513 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.219713 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1491,"thread_start_us":1349,"threads_started":1}
I20260812 16:43:06.220096 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.209551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.31.61.130:42611)
0812 16:43:06.212083 (+ 2532us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.212092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.212119 (+ 27us) server_negotiation.cc:407] Connection header received
0812 16:43:06.212212 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.212219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.212432 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.212629 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:06.213205 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.214586 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.216799 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.217919 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.219360 (+ 1441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.219538 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.219588 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.219764 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2371,"thread_start_us":2257,"threads_started":1}
I20260812 16:43:06.220628 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.209802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:44103 (local address 127.0.0.1:32928)
0812 16:43:06.211322 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:06.211342 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:06.211451 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:06.213945 (+ 2494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:06.213949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:06.213979 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:06.214487 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.214500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.215945 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.215949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.217978 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.217992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.219776 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.219782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.219951 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.219992 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 16:43:06.220085 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 16:43:06.220375 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"thread_start_us":636,"threads_started":1}
I20260812 16:43:06.220659 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.210283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.31.61.129:44103)
0812 16:43:06.213167 (+ 2884us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.213175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.213201 (+ 26us) server_negotiation.cc:407] Connection header received
0812 16:43:06.213284 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.213290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.213511 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.213684 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:06.214695 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.215791 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.218205 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.219567 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.219726 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.219942 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.220143 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.220314 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2726,"thread_start_us":2625,"threads_started":1}
I20260812 16:43:06.223724 17777 tablet_service.cc:1467] Processing CreateTablet for tablet f7b1d7198c954339a158e81bb6bee6f6 (DEFAULT_TABLE table=test-table [id=f8ffcacb48724bb1a5600d4b4661b679]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:06.223883 17908 tablet_service.cc:1467] Processing CreateTablet for tablet f7b1d7198c954339a158e81bb6bee6f6 (DEFAULT_TABLE table=test-table [id=f8ffcacb48724bb1a5600d4b4661b679]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:06.224152 17645 tablet_service.cc:1467] Processing CreateTablet for tablet f7b1d7198c954339a158e81bb6bee6f6 (DEFAULT_TABLE table=test-table [id=f8ffcacb48724bb1a5600d4b4661b679]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:06.224956 17777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b1d7198c954339a158e81bb6bee6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:06.225317 17645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b1d7198c954339a158e81bb6bee6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:06.225314 17908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b1d7198c954339a158e81bb6bee6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:06.243300 17993 tablet_bootstrap.cc:492] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: Bootstrap starting.
I20260812 16:43:06.245563 17992 tablet_bootstrap.cc:492] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: Bootstrap starting.
I20260812 16:43:06.248368 17991 tablet_bootstrap.cc:492] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: Bootstrap starting.
I20260812 16:43:06.249053 17992 tablet_bootstrap.cc:654] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:06.249274 17993 tablet_bootstrap.cc:654] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:06.250216 17993 log.cc:824] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:06.250331 17992 log.cc:824] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:06.251736 17991 tablet_bootstrap.cc:654] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:06.252946 17991 log.cc:824] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:06.253436 17993 tablet_bootstrap.cc:492] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: No bootstrap required, opened a new log
I20260812 16:43:06.253468 17992 tablet_bootstrap.cc:492] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: No bootstrap required, opened a new log
I20260812 16:43:06.253738 17993 ts_tablet_manager.cc:1397] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260812 16:43:06.253741 17992 ts_tablet_manager.cc:1397] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 16:43:06.254087 17710 heartbeater.cc:503] Master 127.31.61.190:36199 was elected leader, sending a full tablet report...
I20260812 16:43:06.262662 17991 tablet_bootstrap.cc:492] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: No bootstrap required, opened a new log
I20260812 16:43:06.262991 17991 ts_tablet_manager.cc:1397] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260812 16:43:06.262729 17992 raft_consensus.cc:348] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } }
I20260812 16:43:06.263139 17992 raft_consensus.cc:374] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:06.263270 17992 raft_consensus.cc:729] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7cdd2f29f2471781f230a3188c4438, State: Initialized, Role: FOLLOWER
I20260812 16:43:06.263908 17992 consensus_queue.cc:260] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } }
I20260812 16:43:06.264228 17993 raft_consensus.cc:348] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } }
I20260812 16:43:06.264750 17993 raft_consensus.cc:374] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:06.264882 17993 raft_consensus.cc:729] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811e176889bd47bb83a074ae6f3756d2, State: Initialized, Role: FOLLOWER
I20260812 16:43:06.265002 17842 heartbeater.cc:503] Master 127.31.61.190:36199 was elected leader, sending a full tablet report...
I20260812 16:43:06.265681 17992 ts_tablet_manager.cc:1428] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260812 16:43:06.265676 17993 consensus_queue.cc:260] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } }
I20260812 16:43:06.267125 17973 heartbeater.cc:503] Master 127.31.61.190:36199 was elected leader, sending a full tablet report...
I20260812 16:43:06.269963 17993 ts_tablet_manager.cc:1428] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
I20260812 16:43:06.272665 17991 raft_consensus.cc:348] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } }
I20260812 16:43:06.273025 17991 raft_consensus.cc:374] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:06.273111 17991 raft_consensus.cc:729] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1264ef4de44f309b16e16ef0dff1a7, State: Initialized, Role: FOLLOWER
I20260812 16:43:06.273625 17991 consensus_queue.cc:260] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } }
I20260812 16:43:06.275130 17991 ts_tablet_manager.cc:1428] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: Time spent starting tablet: real 0.012s user 0.007s sys 0.007s
I20260812 16:43:06.396174 17998 raft_consensus.cc:482] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:43:06.396488 17998 raft_consensus.cc:504] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } }
I20260812 16:43:06.398217 17998 leader_election.cc:288] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b1264ef4de44f309b16e16ef0dff1a7 (127.31.61.129:44103), 5e7cdd2f29f2471781f230a3188c4438 (127.31.61.130:42611)
W20260812 16:43:06.401476 17974 tablet.cc:2369] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:06.405622 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.398743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:39719 (local address 127.31.61.129:44103)
0812 16:43:06.399081 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.399088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.399111 (+ 23us) server_negotiation.cc:407] Connection header received
0812 16:43:06.399187 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.399192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.399282 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.399364 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:06.399897 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.400861 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.403283 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.404575 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.404752 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.404835 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.405208 (+ 373us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.405342 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260812 16:43:06.405646 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.398491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:44103 (local address 127.31.61.131:39719)
0812 16:43:06.398775 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:06.398797 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:06.398939 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:06.399489 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:06.399495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:06.399522 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:06.399726 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.399738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.400989 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.400993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.403107 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.403123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.404781 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.404787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.404988 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.405027 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 16:43:06.405140 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 16:43:06.405284 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1}
I20260812 16:43:06.405668 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.398541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:42611 (local address 127.31.61.131:46603)
0812 16:43:06.399276 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:06.399292 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:06.399419 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:06.399813 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:06.399818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:06.399845 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:06.400153 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.400161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.401391 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.401397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.403383 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:06.403398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.405016 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.405022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:06.405183 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.405212 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 16:43:06.405286 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:43:06.405413 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":401,"threads_started":1}
I20260812 16:43:06.406020 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.398831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:46603 (local address 127.31.61.130:42611)
0812 16:43:06.399273 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.399279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.399368 (+ 89us) server_negotiation.cc:407] Connection header received
0812 16:43:06.399566 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.399571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.399685 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.399767 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:06.400320 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.401240 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.403635 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.404837 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.404990 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.405055 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.405349 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.405494 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260812 16:43:06.406749 17665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b1d7198c954339a158e81bb6bee6f6" candidate_uuid: "811e176889bd47bb83a074ae6f3756d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" is_pre_election: true
I20260812 16:43:06.406822 17797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b1d7198c954339a158e81bb6bee6f6" candidate_uuid: "811e176889bd47bb83a074ae6f3756d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7cdd2f29f2471781f230a3188c4438" is_pre_election: true
I20260812 16:43:06.407274 17797 raft_consensus.cc:2454] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811e176889bd47bb83a074ae6f3756d2 in term 0.
I20260812 16:43:06.407274 17665 raft_consensus.cc:2454] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811e176889bd47bb83a074ae6f3756d2 in term 0.
I20260812 16:43:06.407917 17861 leader_election.cc:302] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1264ef4de44f309b16e16ef0dff1a7, 811e176889bd47bb83a074ae6f3756d2; no voters:
I20260812 16:43:06.408317 17998 raft_consensus.cc:2781] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:43:06.408471 17998 raft_consensus.cc:482] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:43:06.408576 17998 raft_consensus.cc:3037] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:06.411056 17998 raft_consensus.cc:504] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } }
I20260812 16:43:06.411747 17998 leader_election.cc:288] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [CANDIDATE]: Term 1 election: Requested vote from peers 2b1264ef4de44f309b16e16ef0dff1a7 (127.31.61.129:44103), 5e7cdd2f29f2471781f230a3188c4438 (127.31.61.130:42611)
I20260812 16:43:06.412274 17665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b1d7198c954339a158e81bb6bee6f6" candidate_uuid: "811e176889bd47bb83a074ae6f3756d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1264ef4de44f309b16e16ef0dff1a7"
I20260812 16:43:06.412287 17797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b1d7198c954339a158e81bb6bee6f6" candidate_uuid: "811e176889bd47bb83a074ae6f3756d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7cdd2f29f2471781f230a3188c4438"
I20260812 16:43:06.412473 17797 raft_consensus.cc:3037] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:06.412519 17665 raft_consensus.cc:3037] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:06.414745 17797 raft_consensus.cc:2454] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811e176889bd47bb83a074ae6f3756d2 in term 1.
I20260812 16:43:06.414724 17665 raft_consensus.cc:2454] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811e176889bd47bb83a074ae6f3756d2 in term 1.
I20260812 16:43:06.415201 17861 leader_election.cc:302] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b1264ef4de44f309b16e16ef0dff1a7, 5e7cdd2f29f2471781f230a3188c4438, 811e176889bd47bb83a074ae6f3756d2; no voters:
I20260812 16:43:06.415729 17998 raft_consensus.cc:2781] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:06.416164 17998 raft_consensus.cc:686] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 1 LEADER]: Becoming Leader. State: Replica: 811e176889bd47bb83a074ae6f3756d2, State: Running, Role: LEADER
I20260812 16:43:06.416849 17998 consensus_queue.cc:237] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } }
I20260812 16:43:06.421413 17523 catalog_manager.cc:5495] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 811e176889bd47bb83a074ae6f3756d2 (127.31.61.131). New cstate: current_term: 1 leader_uuid: "811e176889bd47bb83a074ae6f3756d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811e176889bd47bb83a074ae6f3756d2" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 } health_report { overall_health: UNKNOWN } } }
W20260812 16:43:06.438824 17843 tablet.cc:2369] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:06.469482 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.463127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.31.61.131:43725)
0812 16:43:06.463329 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.463333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.463351 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:43:06.463455 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.463459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.463540 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.463717 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:43:06.464214 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.465191 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.466893 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.467417 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.467529 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.468222 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:43:06.468888 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:43:06.469047 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.469203 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.469278 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 16:43:06.478246 17665 raft_consensus.cc:1261] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [term 1 FOLLOWER]: Refusing update from remote peer 811e176889bd47bb83a074ae6f3756d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:06.478246 17797 raft_consensus.cc:1261] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [term 1 FOLLOWER]: Refusing update from remote peer 811e176889bd47bb83a074ae6f3756d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:06.479441 18001 consensus_queue.cc:1035] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7cdd2f29f2471781f230a3188c4438" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 42611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:06.480131 17998 consensus_queue.cc:1035] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1264ef4de44f309b16e16ef0dff1a7" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 44103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:43:06.480952 17711 tablet.cc:2369] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:06.505183 17522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:06.505465 17522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:06.509413 17522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca879d154f4b40a588d204f46c767c42: Sending DeleteTablet for 3 replicas of tablet f7b1d7198c954339a158e81bb6bee6f6
I20260812 16:43:06.510973 17777 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b1d7198c954339a158e81bb6bee6f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:06 UTC) from {username='slave'} at 127.0.0.1:57744
I20260812 16:43:06.510974 17645 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b1d7198c954339a158e81bb6bee6f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:06 UTC) from {username='slave'} at 127.0.0.1:32928
I20260812 16:43:06.511972 17908 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b1d7198c954339a158e81bb6bee6f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:06 UTC) from {username='slave'} at 127.0.0.1:39426
I20260812 16:43:06.512157 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 17582
I20260812 16:43:06.512303 18015 tablet_replica.cc:330] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: stopping tablet replica
I20260812 16:43:06.512795 18015 raft_consensus.cc:2229] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:06.513355 18015 raft_consensus.cc:2258] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:06.514048 18014 tablet_replica.cc:330] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: stopping tablet replica
I20260812 16:43:06.514567 18014 raft_consensus.cc:2229] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:06.515111 18014 raft_consensus.cc:2258] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:06.517525 18014 ts_tablet_manager.cc:1905] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:06.517524 18015 ts_tablet_manager.cc:1905] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:06.525108 18015 ts_tablet_manager.cc:1918] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:06.525398 18014 ts_tablet_manager.cc:1918] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:06.525714 18014 log.cc:1196] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/f7b1d7198c954339a158e81bb6bee6f6
I20260812 16:43:06.526383 18014 ts_tablet_manager.cc:1939] T f7b1d7198c954339a158e81bb6bee6f6 P 2b1264ef4de44f309b16e16ef0dff1a7: Deleting consensus metadata
I20260812 16:43:06.528548 18015 log.cc:1196] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/f7b1d7198c954339a158e81bb6bee6f6
I20260812 16:43:06.528656 17508 catalog_manager.cc:4841] TS 2b1264ef4de44f309b16e16ef0dff1a7 (127.31.61.129:44103): tablet f7b1d7198c954339a158e81bb6bee6f6 (table test-table [id=f8ffcacb48724bb1a5600d4b4661b679]) successfully deleted
I20260812 16:43:06.529241 18015 ts_tablet_manager.cc:1939] T f7b1d7198c954339a158e81bb6bee6f6 P 5e7cdd2f29f2471781f230a3188c4438: Deleting consensus metadata
I20260812 16:43:06.531215 17509 catalog_manager.cc:4841] TS 5e7cdd2f29f2471781f230a3188c4438 (127.31.61.130:42611): tablet f7b1d7198c954339a158e81bb6bee6f6 (table test-table [id=f8ffcacb48724bb1a5600d4b4661b679]) successfully deleted
I20260812 16:43:06.531539 18016 tablet_replica.cc:330] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: stopping tablet replica
I20260812 16:43:06.531963 18016 raft_consensus.cc:2229] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:43:06.532519 18016 raft_consensus.cc:2258] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:06.533790 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.513545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.31.61.129:44103)
0812 16:43:06.513727 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.513733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.524216 (+ 10483us) server_negotiation.cc:407] Connection header received
0812 16:43:06.524468 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.524475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.524578 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.524647 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:06.525364 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.526556 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.529271 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.529915 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.530019 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.532100 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:06.532112 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:06.532130 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:06.532883 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:06.532966 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:06.532993 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:06.533099 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.533308 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.533429 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:43:06.534348 18016 ts_tablet_manager.cc:1905] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:06.534948 17705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:06.540735 18016 ts_tablet_manager.cc:1918] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:06.540895 18016 log.cc:1196] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/f7b1d7198c954339a158e81bb6bee6f6
I20260812 16:43:06.541373 18016 ts_tablet_manager.cc:1939] T f7b1d7198c954339a158e81bb6bee6f6 P 811e176889bd47bb83a074ae6f3756d2: Deleting consensus metadata
I20260812 16:43:06.542833 17508 catalog_manager.cc:4841] TS 811e176889bd47bb83a074ae6f3756d2 (127.31.61.131:43725): tablet f7b1d7198c954339a158e81bb6bee6f6 (table test-table [id=f8ffcacb48724bb1a5600d4b4661b679]) successfully deleted
I20260812 16:43:06.681185 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 17582
I20260812 16:43:06.702771 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 17714
I20260812 16:43:06.711514 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.703858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.31.61.130:42611)
0812 16:43:06.704112 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.704119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.704140 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:43:06.704204 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.704209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.704308 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.704430 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:06.704887 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.706103 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.707966 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.708644 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.708773 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.709942 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:06.709963 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:06.709982 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:06.710726 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:06.710788 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:06.710813 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:06.710930 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.711073 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.711187 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260812 16:43:06.712251 17837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:06.802250 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 17714
I20260812 16:43:06.817996 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 17845
I20260812 16:43:06.825670 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:06.818928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.31.61.131:43725)
0812 16:43:06.819147 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:06.819154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:06.819175 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:43:06.819247 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:06.819251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:06.819335 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:06.819451 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:06.819924 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.820806 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.822367 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:06.822837 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:06.822966 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:06.824244 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:06.824257 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:06.824273 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:06.824890 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:06.824938 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:06.824961 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:06.825076 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:06.825213 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 16:43:06.825318 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:43:06.826413 17968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:06.945219 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 17845
I20260812 16:43:06.962347 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 17491
I20260812 16:43:06.963529 17553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:07.079707 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 17491
2026-08-12T16:43:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2961 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:07 dist-test-slave-f7th krb5kdc[18024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:07 dist-test-slave-f7th krb5kdc[18024](info): set up 2 sockets
Aug 12 16:43:07 dist-test-slave-f7th krb5kdc[18024](info): commencing operation
krb5kdc: starting...
W20260812 16:43:08.139216 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:08 dist-test-slave-f7th krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:08.416942 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:45271
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:45271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:08.752853 18037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:08.753165 18037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:08.753259 18037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:08.762205 18037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:08.762346 18037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:08.762398 18037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:08.762456 18037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:08.762501 18037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:08.774245 18037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:45271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:45271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:08.775691 18037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:08.777088 18037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:08.790390 18045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:08.790458 18042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:08.790491 18043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:08.790880 18037 server_base.cc:1034] running on GCE node
I20260812 16:43:08.792112 18037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:08.793670 18037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:08.794931 18037 hybrid_clock.cc:648] HybridClock initialized: now 1786552988794847 us; error 93 us; skew 500 ppm
I20260812 16:43:08.795354 18037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:08 dist-test-slave-f7th krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552988, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:08.799038 18037 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:08.800839 18037 webserver.cc:460] Webserver started at http://127.31.61.190:46063/ using document root <none> and password file <none>
I20260812 16:43:08.801867 18037 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:08.802021 18037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:08.802356 18037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:08.805513 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "1962ba572a324e4f90722eada5bee6a5"
format_stamp: "Formatted at 2026-08-12 16:43:08 on dist-test-slave-f7th"
server_key: "823258bdb5e77f0cd9f039218ac210cf"
server_key_iv: "be513d067fe55babb0dc428edf65bc8e"
server_key_version: "encryptionkey@0"
I20260812 16:43:08.806270 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "1962ba572a324e4f90722eada5bee6a5"
format_stamp: "Formatted at 2026-08-12 16:43:08 on dist-test-slave-f7th"
server_key: "823258bdb5e77f0cd9f039218ac210cf"
server_key_iv: "be513d067fe55babb0dc428edf65bc8e"
server_key_version: "encryptionkey@0"
I20260812 16:43:08.811682 18037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:43:08.815320 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:08.816797 18037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 16:43:08.816984 18037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "1962ba572a324e4f90722eada5bee6a5"
format_stamp: "Formatted at 2026-08-12 16:43:08 on dist-test-slave-f7th"
server_key: "823258bdb5e77f0cd9f039218ac210cf"
server_key_iv: "be513d067fe55babb0dc428edf65bc8e"
server_key_version: "encryptionkey@0"
I20260812 16:43:08.817176 18037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:08.843163 18037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:08.847733 18037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:08.848067 18037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:08.868137 18037 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:45271
I20260812 16:43:08.868193 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:45271 every 8 connection(s)
I20260812 16:43:08.869551 18037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:08.872876 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 18037
I20260812 16:43:08.873097 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:08.873534 31990 external_mini_cluster.cc:1428] Setting key a81872979fcd5526f3da130ba0e83ae5
I20260812 16:43:08.873713 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:08.888652 18104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5: Bootstrap starting.
Aug 12 16:43:08 dist-test-slave-f7th krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:08.892163 18104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:08.893239 18104 log.cc:824] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:08.896113 18104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5: No bootstrap required, opened a new log
I20260812 16:43:08.897441 18107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:08.875376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.31.61.190:45271)
0812 16:43:08.876215 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:08.876227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:08.876267 (+ 40us) server_negotiation.cc:407] Connection header received
0812 16:43:08.878541 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:08.878567 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:08.879001 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:08.879335 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:08.881271 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:08.882590 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:08.884469 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:08.885199 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:08.885354 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:08.890289 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:08.890299 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:08.890315 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:08.890365 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:08.893470 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:08.894180 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:08.894188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:08.894198 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:08.894377 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:08.894743 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:08.894747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:08.894749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:08.895267 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:08.895563 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:08.896035 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 16:43:08.896483 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":405,"threads_started":1}
I20260812 16:43:08.904026 18104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1962ba572a324e4f90722eada5bee6a5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45271 } }
I20260812 16:43:08.904464 18104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:08.904561 18104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1962ba572a324e4f90722eada5bee6a5, State: Initialized, Role: FOLLOWER
I20260812 16:43:08.905157 18104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1962ba572a324e4f90722eada5bee6a5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45271 } }
I20260812 16:43:08.905397 18104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:08.905539 18104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:08.905699 18104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:08.908018 18104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1962ba572a324e4f90722eada5bee6a5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45271 } }
I20260812 16:43:08.908545 18104 leader_election.cc:302] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1962ba572a324e4f90722eada5bee6a5; no voters:
I20260812 16:43:08.909205 18104 leader_election.cc:288] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:08.909327 18109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:08.909979 18109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [term 1 LEADER]: Becoming Leader. State: Replica: 1962ba572a324e4f90722eada5bee6a5, State: Running, Role: LEADER
I20260812 16:43:08.910483 18109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1962ba572a324e4f90722eada5bee6a5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45271 } }
I20260812 16:43:08.911240 18104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:08.913795 18111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1962ba572a324e4f90722eada5bee6a5. Latest consensus state: current_term: 1 leader_uuid: "1962ba572a324e4f90722eada5bee6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1962ba572a324e4f90722eada5bee6a5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45271 } } }
I20260812 16:43:08.914191 18111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:08.913764 18110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1962ba572a324e4f90722eada5bee6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1962ba572a324e4f90722eada5bee6a5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 45271 } } }
I20260812 16:43:08.914693 18110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:08.916390 18115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:08.921645 18115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:08.931164 18115 catalog_manager.cc:1347] Generated new cluster ID: d99a0a5b7e024059971afebab34dc3e8
I20260812 16:43:08.931315 18115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:08.940384 18115 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:08.942279 18115 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:08.955713 18115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5: Generated new TSK 0
I20260812 16:43:08.956881 18115 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:43:09.037220 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45271
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:09.390455 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:09.390734 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:09.390828 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:09.398607 18131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:09.398736 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:09.398859 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:43:09.409482 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:09.410795 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:09.412263 18131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:09.428251 18136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:09.428143 18139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:09.428289 18137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:09.428896 18131 server_base.cc:1034] running on GCE node
I20260812 16:43:09.429665 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:09.430919 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:09.432118 18131 hybrid_clock.cc:648] HybridClock initialized: now 1786552989432069 us; error 45 us; skew 500 ppm
I20260812 16:43:09.432463 18131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:09 dist-test-slave-f7th krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:09.435539 18131 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:43:09.437057 18131 webserver.cc:460] Webserver started at http://127.31.61.129:34369/ using document root <none> and password file <none>
I20260812 16:43:09.437736 18131 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:09.437855 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:09.438155 18131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:09.441315 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "ef8c4f54f38345e48b78734880eff690"
format_stamp: "Formatted at 2026-08-12 16:43:09 on dist-test-slave-f7th"
server_key: "d5ca374809054cf1e8ca7065c1317dc4"
server_key_iv: "54a2b7ca1deea946df7bad3dbb20cb8a"
server_key_version: "encryptionkey@0"
I20260812 16:43:09.442147 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "ef8c4f54f38345e48b78734880eff690"
format_stamp: "Formatted at 2026-08-12 16:43:09 on dist-test-slave-f7th"
server_key: "d5ca374809054cf1e8ca7065c1317dc4"
server_key_iv: "54a2b7ca1deea946df7bad3dbb20cb8a"
server_key_version: "encryptionkey@0"
I20260812 16:43:09.447353 18131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:43:09.450814 18146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:09.452149 18131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 16:43:09.452301 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "ef8c4f54f38345e48b78734880eff690"
format_stamp: "Formatted at 2026-08-12 16:43:09 on dist-test-slave-f7th"
server_key: "d5ca374809054cf1e8ca7065c1317dc4"
server_key_iv: "54a2b7ca1deea946df7bad3dbb20cb8a"
server_key_version: "encryptionkey@0"
I20260812 16:43:09.452500 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:09.476733 18131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:09.480850 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:09.481165 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:09.482194 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:09.483618 18131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:09.483695 18131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:09.483814 18131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:09.483866 18131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:09.514922 18131 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:38227
I20260812 16:43:09.514979 18260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:38227 every 8 connection(s)
I20260812 16:43:09.516251 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:43:09.524046 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 18131
I20260812 16:43:09.524248 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:43:09.524797 31990 external_mini_cluster.cc:1428] Setting key ffe01d62232f66dbc2e05a4feb1b57ee
Aug 12 16:43:09 dist-test-slave-f7th krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:09.537665 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:09.518819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:32949 (local address 127.31.61.190:45271)
0812 16:43:09.519325 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:09.519331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:09.520350 (+ 1019us) server_negotiation.cc:407] Connection header received
0812 16:43:09.522183 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:09.522188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:09.522259 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:09.522391 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:09.524051 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:09.525126 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:09.526906 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:09.527487 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:09.527627 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:09.531214 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:09.531221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:09.531225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:09.531265 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:09.534257 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:09.535017 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:09.535022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:09.535025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:09.535166 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:09.535722 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:09.535727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:09.535729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:09.536069 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:09.536212 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:09.537190 (+ 978us) server_negotiation.cc:299] Negotiation successful
0812 16:43:09.537398 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":234,"threads_started":1}
I20260812 16:43:09.539152 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:09.519423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45271 (local address 127.31.61.129:32949)
0812 16:43:09.520187 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:09.520234 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:09.521948 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:09.522672 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:09.522680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:09.523222 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:09.523868 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:09.523888 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:09.525281 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:09.525285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:09.526750 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:09.526763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:09.527752 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:09.527758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:09.527958 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:09.528626 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:09.528649 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:09.530948 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:09.534411 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:09.534419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:09.534432 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:09.534887 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:09.535312 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:09.535317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:09.535329 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:09.535567 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:09.536217 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:09.536227 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:09.536712 (+ 485us) client_negotiation.cc:769] Sending connection context
0812 16:43:09.537061 (+ 349us) client_negotiation.cc:240] Negotiation successful
0812 16:43:09.537657 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":313,"threads_started":1}
I20260812 16:43:09.540865 18261 heartbeater.cc:347] Connected to a master server at 127.31.61.190:45271
I20260812 16:43:09.541198 18261 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:09.541919 18261 heartbeater.cc:511] Master 127.31.61.190:45271 requested a full tablet report, sending...
I20260812 16:43:09.544159 18069 ts_manager.cc:194] Registered new tserver with Master: ef8c4f54f38345e48b78734880eff690 (127.31.61.129:38227)
I20260812 16:43:09.546005 18069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:32949
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:43:09.583132 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45271
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:09.911808 18267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:09.912103 18267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:09.912211 18267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:09.919584 18267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:09.919731 18267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:09.919852 18267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:43:09.929930 18267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:09.931268 18267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:09.932648 18267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:09.946034 18272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:09.946063 18273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:09.946112 18275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:09.947273 18267 server_base.cc:1034] running on GCE node
I20260812 16:43:09.948067 18267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:09.949115 18267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:09.950340 18267 hybrid_clock.cc:648] HybridClock initialized: now 1786552989950281 us; error 43 us; skew 500 ppm
I20260812 16:43:09.950681 18267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:09 dist-test-slave-f7th krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:09.958254 18267 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:43:09.960007 18267 webserver.cc:460] Webserver started at http://127.31.61.130:37103/ using document root <none> and password file <none>
I20260812 16:43:09.960714 18267 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:09.960826 18267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:09.961129 18267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:09.964382 18267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "4edc8a1b568b462f938bee8dd2cf61ce"
format_stamp: "Formatted at 2026-08-12 16:43:09 on dist-test-slave-f7th"
server_key: "0e5642b52efe50b8ba2f0b00d3f18744"
server_key_iv: "eea38a104e076aa8634562da1a164890"
server_key_version: "encryptionkey@0"
I20260812 16:43:09.965214 18267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "4edc8a1b568b462f938bee8dd2cf61ce"
format_stamp: "Formatted at 2026-08-12 16:43:09 on dist-test-slave-f7th"
server_key: "0e5642b52efe50b8ba2f0b00d3f18744"
server_key_iv: "eea38a104e076aa8634562da1a164890"
server_key_version: "encryptionkey@0"
I20260812 16:43:09.970783 18267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260812 16:43:09.974674 18282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:09.976238 18267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:09.976509 18267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "4edc8a1b568b462f938bee8dd2cf61ce"
format_stamp: "Formatted at 2026-08-12 16:43:09 on dist-test-slave-f7th"
server_key: "0e5642b52efe50b8ba2f0b00d3f18744"
server_key_iv: "eea38a104e076aa8634562da1a164890"
server_key_version: "encryptionkey@0"
I20260812 16:43:09.976734 18267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:10.019757 18267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:10.024376 18267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:10.024677 18267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:10.025622 18267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:10.026993 18267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:10.027066 18267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:10.027153 18267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:10.027199 18267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:10.058982 18267 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:39149
I20260812 16:43:10.059032 18395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:39149 every 8 connection(s)
I20260812 16:43:10.060267 18267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:43:10.068454 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 18267
I20260812 16:43:10.068658 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:43:10.069252 31990 external_mini_cluster.cc:1428] Setting key 247c689f04d47a929005212af9dbad6e
Aug 12 16:43:10 dist-test-slave-f7th krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:10.082267 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.062862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:47269 (local address 127.31.61.190:45271)
0812 16:43:10.063315 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:10.063320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:10.064306 (+ 986us) server_negotiation.cc:407] Connection header received
0812 16:43:10.066191 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:10.066197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:10.066288 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:10.066386 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:10.067961 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.068676 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.070855 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.071479 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.071656 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.076115 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:10.076121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:10.076124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:10.076162 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:10.078789 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:10.079732 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:10.079737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:10.079739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:10.079899 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:10.080407 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:10.080411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:10.080413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:10.080718 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:10.080833 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:10.081639 (+ 806us) server_negotiation.cc:299] Negotiation successful
0812 16:43:10.081891 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260812 16:43:10.083261 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.063410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45271 (local address 127.31.61.130:47269)
0812 16:43:10.064132 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:10.064182 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:10.065911 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:10.066646 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:10.066655 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:10.067166 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:10.067779 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.067798 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.068871 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.068878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.070680 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.070697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.071787 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.071795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.072104 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.072847 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:10.072877 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:10.075821 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:10.078956 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:10.078966 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:10.078987 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:10.079580 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:10.080008 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:10.080012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:10.080015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:10.080279 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:10.080829 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:10.080836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:10.081243 (+ 407us) client_negotiation.cc:769] Sending connection context
0812 16:43:10.081494 (+ 251us) client_negotiation.cc:240] Negotiation successful
0812 16:43:10.082057 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":293,"threads_started":1}
I20260812 16:43:10.084704 18396 heartbeater.cc:347] Connected to a master server at 127.31.61.190:45271
I20260812 16:43:10.085032 18396 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:10.085747 18396 heartbeater.cc:511] Master 127.31.61.190:45271 requested a full tablet report, sending...
I20260812 16:43:10.087191 18069 ts_manager.cc:194] Registered new tserver with Master: 4edc8a1b568b462f938bee8dd2cf61ce (127.31.61.130:39149)
I20260812 16:43:10.088276 18069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:47269
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:43:10.131327 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45271
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:10.465987 18402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:10.466279 18402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:10.466374 18402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:10.473788 18402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:10.473912 18402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:10.474059 18402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:43:10.484512 18402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:45271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:10.485852 18402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:10.487136 18402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:10.500170 18408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:10.500131 18407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:10.500097 18410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:10.505148 18402 server_base.cc:1034] running on GCE node
I20260812 16:43:10.505896 18402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:10.506917 18402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:10.508131 18402 hybrid_clock.cc:648] HybridClock initialized: now 1786552990508077 us; error 56 us; skew 500 ppm
I20260812 16:43:10.508482 18402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:10 dist-test-slave-f7th krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:10.511587 18402 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:43:10.513154 18402 webserver.cc:460] Webserver started at http://127.31.61.131:38647/ using document root <none> and password file <none>
I20260812 16:43:10.513842 18402 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:10.513957 18402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:10.514297 18402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:10.518224 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "161dbd09d60147059f4ade5a22f0a26d"
format_stamp: "Formatted at 2026-08-12 16:43:10 on dist-test-slave-f7th"
server_key: "d45d867b437c3e3bcb9e4bba6f1bbd65"
server_key_iv: "f64405ae1c0fb1765c7bb3f0b34c5899"
server_key_version: "encryptionkey@0"
I20260812 16:43:10.519205 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "161dbd09d60147059f4ade5a22f0a26d"
format_stamp: "Formatted at 2026-08-12 16:43:10 on dist-test-slave-f7th"
server_key: "d45d867b437c3e3bcb9e4bba6f1bbd65"
server_key_iv: "f64405ae1c0fb1765c7bb3f0b34c5899"
server_key_version: "encryptionkey@0"
I20260812 16:43:10.524823 18402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:43:10.528174 18417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:10.529456 18402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:43:10.529592 18402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "161dbd09d60147059f4ade5a22f0a26d"
format_stamp: "Formatted at 2026-08-12 16:43:10 on dist-test-slave-f7th"
server_key: "d45d867b437c3e3bcb9e4bba6f1bbd65"
server_key_iv: "f64405ae1c0fb1765c7bb3f0b34c5899"
server_key_version: "encryptionkey@0"
I20260812 16:43:10.529767 18402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:10.547673 18402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:10.549194 18261 heartbeater.cc:503] Master 127.31.61.190:45271 was elected leader, sending a full tablet report...
I20260812 16:43:10.552418 18402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:10.552701 18402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:10.553671 18402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:10.555061 18402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:10.555140 18402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:10.555262 18402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:10.555315 18402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:10.586715 18402 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:46449
I20260812 16:43:10.586755 18532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:46449 every 8 connection(s)
I20260812 16:43:10.588101 18402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:43:10.596912 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 18402
I20260812 16:43:10.597107 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:43:10.597573 31990 external_mini_cluster.cc:1428] Setting key fe77ac5169561411e1b461904531974f
Aug 12 16:43:10 dist-test-slave-f7th krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:10.607643 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.590499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:59823 (local address 127.31.61.190:45271)
0812 16:43:10.590967 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:10.590972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:10.591792 (+ 820us) server_negotiation.cc:407] Connection header received
0812 16:43:10.593562 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:10.593568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:10.593648 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:10.593766 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:10.595220 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.596237 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.597941 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.598460 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.598603 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.602052 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:10.602057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:10.602060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:10.602095 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:10.604623 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:10.605323 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:10.605329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:10.605332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:10.605456 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:10.605864 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:10.605869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:10.605872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:10.606144 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:10.606289 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:10.606978 (+ 689us) server_negotiation.cc:299] Negotiation successful
0812 16:43:10.607213 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":228,"threads_started":1}
I20260812 16:43:10.608610 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.590938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:45271 (local address 127.31.61.131:59823)
0812 16:43:10.591640 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:10.591690 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:10.593311 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:10.594005 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:10.594016 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:10.594502 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:10.595053 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.595070 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.596419 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.596423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.597790 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.597803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.598721 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.598731 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.598960 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.599579 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:10.599653 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:10.601830 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:10.604755 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:10.604765 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:10.604783 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:10.605207 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:10.605560 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:10.605565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:10.605569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:10.605758 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:10.606270 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:10.606279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:10.606628 (+ 349us) client_negotiation.cc:769] Sending connection context
0812 16:43:10.606884 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 16:43:10.607442 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":245,"threads_started":1}
I20260812 16:43:10.609970 18533 heartbeater.cc:347] Connected to a master server at 127.31.61.190:45271
I20260812 16:43:10.610250 18533 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:10.611001 18533 heartbeater.cc:511] Master 127.31.61.190:45271 requested a full tablet report, sending...
I20260812 16:43:10.612854 18068 ts_manager.cc:194] Registered new tserver with Master: 161dbd09d60147059f4ade5a22f0a26d (127.31.61.131:46449)
I20260812 16:43:10.613745 18068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:59823
I20260812 16:43:10.625125 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:43:10.641637 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.629027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.31.61.190:45271)
0812 16:43:10.629176 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:10.629180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:10.629574 (+ 394us) server_negotiation.cc:407] Connection header received
0812 16:43:10.629718 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:10.629722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:10.629784 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:10.629864 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:10.631246 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.632087 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.633414 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.633915 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.634028 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.636730 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:10.636736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:10.636739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:10.636774 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:10.639205 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:10.639974 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:10.639978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:10.639980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:10.640108 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:10.640473 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:10.640476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:10.640478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:10.640775 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:10.640975 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:10.641255 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 16:43:10.641395 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 16:43:10.645764 18068 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:43:10.652623 18068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:43:10.676375 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.666204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.31.61.131:46449)
0812 16:43:10.666765 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:10.666771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:10.666800 (+ 29us) server_negotiation.cc:407] Connection header received
0812 16:43:10.667139 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:10.667147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:10.667342 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:10.667536 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:10.668323 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.669977 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.673042 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.674595 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.674706 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.674965 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:10.675745 (+ 780us) server_negotiation.cc:299] Negotiation successful
0812 16:43:10.675961 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
I20260812 16:43:10.677807 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.667079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38227 (local address 127.0.0.1:44632)
0812 16:43:10.668513 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:10.668535 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:10.668643 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:10.669599 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:10.669604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:10.669633 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:10.670062 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.670075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.672145 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.672151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.674481 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.674500 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.676850 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.676857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.677077 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.677113 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 16:43:10.677203 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 16:43:10.677354 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"thread_start_us":1209,"threads_started":1}
I20260812 16:43:10.678092 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.665966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:46449 (local address 127.0.0.1:43288)
0812 16:43:10.666646 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:10.666694 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:10.666858 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:10.667560 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:10.667567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:10.667624 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:10.668075 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.668090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.670200 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.670207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.672808 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.672826 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.675177 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.675184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.675416 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.675467 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 16:43:10.676927 (+ 1460us) client_negotiation.cc:240] Negotiation successful
0812 16:43:10.677185 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":302,"threads_started":1}
I20260812 16:43:10.678714 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.667389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.31.61.129:38227)
0812 16:43:10.668280 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:10.668287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:10.668764 (+ 477us) server_negotiation.cc:407] Connection header received
0812 16:43:10.668876 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:10.668885 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:10.669136 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:10.669319 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:10.670254 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.671950 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.674697 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.676310 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.676451 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.676689 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:10.678309 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0812 16:43:10.678477 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":538,"threads_started":1}
I20260812 16:43:10.684316 18195 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc4ae7fec1a407b9fe8e48197859dc2 (DEFAULT_TABLE table=test-table [id=75a08a89b02446d99a8effd9501b5804]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:10.684233 18466 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc4ae7fec1a407b9fe8e48197859dc2 (DEFAULT_TABLE table=test-table [id=75a08a89b02446d99a8effd9501b5804]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:10.685981 18195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc4ae7fec1a407b9fe8e48197859dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:10.686000 18466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc4ae7fec1a407b9fe8e48197859dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:10.687230 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.668615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:39149 (local address 127.0.0.1:60048)
0812 16:43:10.678219 (+ 9604us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:10.678238 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:10.678369 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:10.679165 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:10.679169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:10.679195 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:10.679625 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.679639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.681771 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.681777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.684217 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.684236 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.686398 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.686405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.686576 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.686614 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 16:43:10.686703 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 16:43:10.686848 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9441,"thread_start_us":9306,"threads_started":1}
I20260812 16:43:10.687992 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.668856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.31.61.130:39149)
0812 16:43:10.677816 (+ 8960us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:10.677826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:10.678423 (+ 597us) server_negotiation.cc:407] Connection header received
0812 16:43:10.678540 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:10.678549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:10.678933 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:10.679162 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:10.679874 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.681586 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.684443 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.685883 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.686029 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.686249 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:10.687494 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0812 16:43:10.687711 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8797,"thread_start_us":8696,"threads_started":1}
I20260812 16:43:10.693884 18331 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc4ae7fec1a407b9fe8e48197859dc2 (DEFAULT_TABLE table=test-table [id=75a08a89b02446d99a8effd9501b5804]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:10.695492 18331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc4ae7fec1a407b9fe8e48197859dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:10.700058 18552 tablet_bootstrap.cc:492] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: Bootstrap starting.
I20260812 16:43:10.704427 18552 tablet_bootstrap.cc:654] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:10.705543 18552 log.cc:824] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:10.708655 18552 tablet_bootstrap.cc:492] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: No bootstrap required, opened a new log
I20260812 16:43:10.708994 18552 ts_tablet_manager.cc:1397] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 16:43:10.709141 18553 tablet_bootstrap.cc:492] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: Bootstrap starting.
I20260812 16:43:10.713333 18553 tablet_bootstrap.cc:654] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:10.714027 18555 tablet_bootstrap.cc:492] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: Bootstrap starting.
I20260812 16:43:10.714809 18553 log.cc:824] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:10.717778 18555 tablet_bootstrap.cc:654] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:10.718701 18553 tablet_bootstrap.cc:492] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: No bootstrap required, opened a new log
I20260812 16:43:10.719004 18553 ts_tablet_manager.cc:1397] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260812 16:43:10.719156 18555 log.cc:824] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:10.720311 18552 raft_consensus.cc:348] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } }
I20260812 16:43:10.720803 18552 raft_consensus.cc:374] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:10.720920 18552 raft_consensus.cc:729] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161dbd09d60147059f4ade5a22f0a26d, State: Initialized, Role: FOLLOWER
I20260812 16:43:10.721570 18552 consensus_queue.cc:260] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } }
I20260812 16:43:10.722522 18555 tablet_bootstrap.cc:492] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: No bootstrap required, opened a new log
I20260812 16:43:10.722687 18533 heartbeater.cc:503] Master 127.31.61.190:45271 was elected leader, sending a full tablet report...
I20260812 16:43:10.722872 18555 ts_tablet_manager.cc:1397] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260812 16:43:10.723322 18552 ts_tablet_manager.cc:1428] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260812 16:43:10.729940 18553 raft_consensus.cc:348] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } }
I20260812 16:43:10.730443 18553 raft_consensus.cc:374] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:10.730548 18553 raft_consensus.cc:729] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8c4f54f38345e48b78734880eff690, State: Initialized, Role: FOLLOWER
I20260812 16:43:10.731202 18553 consensus_queue.cc:260] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } }
I20260812 16:43:10.733059 18553 ts_tablet_manager.cc:1428] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: Time spent starting tablet: real 0.014s user 0.005s sys 0.010s
I20260812 16:43:10.733469 18555 raft_consensus.cc:348] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } }
I20260812 16:43:10.733886 18555 raft_consensus.cc:374] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:10.733987 18555 raft_consensus.cc:729] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edc8a1b568b462f938bee8dd2cf61ce, State: Initialized, Role: FOLLOWER
I20260812 16:43:10.734607 18555 consensus_queue.cc:260] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } }
I20260812 16:43:10.735692 18396 heartbeater.cc:503] Master 127.31.61.190:45271 was elected leader, sending a full tablet report...
I20260812 16:43:10.736230 18555 ts_tablet_manager.cc:1428] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260812 16:43:10.768903 18262 tablet.cc:2369] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:43:10.812402 18397 tablet.cc:2369] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:43:10.839712 18534 tablet.cc:2369] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:10.909567 18560 raft_consensus.cc:482] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:43:10.909886 18560 raft_consensus.cc:504] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } }
I20260812 16:43:10.911651 18560 leader_election.cc:288] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 161dbd09d60147059f4ade5a22f0a26d (127.31.61.131:46449), ef8c4f54f38345e48b78734880eff690 (127.31.61.129:38227)
I20260812 16:43:10.918589 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.912097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:59437 (local address 127.31.61.131:46449)
0812 16:43:10.912341 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:10.912350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:10.912849 (+ 499us) server_negotiation.cc:407] Connection header received
0812 16:43:10.913201 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:10.913207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:10.913290 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:10.913358 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:10.914151 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.914898 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.916833 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.917669 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.917824 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.917903 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:10.918214 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 16:43:10.918344 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 16:43:10.918691 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.913204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:39411 (local address 127.31.61.129:38227)
0812 16:43:10.913408 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:10.913414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:10.913486 (+ 72us) server_negotiation.cc:407] Connection header received
0812 16:43:10.913638 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:10.913643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:10.913719 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:10.913810 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:10.914377 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.915084 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.916923 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.917782 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.917948 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.918021 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:10.918317 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 16:43:10.918452 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:43:10.918744 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.912039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:46449 (local address 127.31.61.130:59437)
0812 16:43:10.912722 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:10.912740 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:10.912899 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:10.913581 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:10.913585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:10.913612 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:10.914010 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.914024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.915038 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.915044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.916685 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.916698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.917869 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.917875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.918058 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.918083 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 16:43:10.918172 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 16:43:10.918300 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":23,"thread_start_us":402,"threads_started":1}
I20260812 16:43:10.918869 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.912848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:38227 (local address 127.31.61.130:39411)
0812 16:43:10.913393 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:10.913406 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:10.913508 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:10.913827 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:10.913830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:10.913854 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:10.914195 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.914203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.915222 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.915226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.916765 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:10.916777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.918025 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.918031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:10.918176 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.918199 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 16:43:10.918274 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 16:43:10.918391 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":29,"thread_start_us":265,"threads_started":1}
I20260812 16:43:10.919813 18488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc4ae7fec1a407b9fe8e48197859dc2" candidate_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161dbd09d60147059f4ade5a22f0a26d" is_pre_election: true
I20260812 16:43:10.919881 18215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc4ae7fec1a407b9fe8e48197859dc2" candidate_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8c4f54f38345e48b78734880eff690" is_pre_election: true
I20260812 16:43:10.920300 18488 raft_consensus.cc:2454] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4edc8a1b568b462f938bee8dd2cf61ce in term 0.
I20260812 16:43:10.920338 18215 raft_consensus.cc:2454] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4edc8a1b568b462f938bee8dd2cf61ce in term 0.
I20260812 16:43:10.920943 18283 leader_election.cc:302] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161dbd09d60147059f4ade5a22f0a26d, 4edc8a1b568b462f938bee8dd2cf61ce; no voters:
I20260812 16:43:10.921353 18560 raft_consensus.cc:2781] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:43:10.921512 18560 raft_consensus.cc:482] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:43:10.921614 18560 raft_consensus.cc:3037] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:10.924043 18560 raft_consensus.cc:504] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } }
I20260812 16:43:10.924612 18560 leader_election.cc:288] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [CANDIDATE]: Term 1 election: Requested vote from peers 161dbd09d60147059f4ade5a22f0a26d (127.31.61.131:46449), ef8c4f54f38345e48b78734880eff690 (127.31.61.129:38227)
I20260812 16:43:10.925060 18488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc4ae7fec1a407b9fe8e48197859dc2" candidate_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161dbd09d60147059f4ade5a22f0a26d"
I20260812 16:43:10.925144 18215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc4ae7fec1a407b9fe8e48197859dc2" candidate_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8c4f54f38345e48b78734880eff690"
I20260812 16:43:10.925261 18488 raft_consensus.cc:3037] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:10.925341 18215 raft_consensus.cc:3037] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:10.927536 18215 raft_consensus.cc:2454] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edc8a1b568b462f938bee8dd2cf61ce in term 1.
I20260812 16:43:10.927538 18488 raft_consensus.cc:2454] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edc8a1b568b462f938bee8dd2cf61ce in term 1.
I20260812 16:43:10.927933 18283 leader_election.cc:302] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edc8a1b568b462f938bee8dd2cf61ce, ef8c4f54f38345e48b78734880eff690; no voters:
I20260812 16:43:10.928184 18560 raft_consensus.cc:2781] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:10.928596 18560 raft_consensus.cc:686] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 1 LEADER]: Becoming Leader. State: Replica: 4edc8a1b568b462f938bee8dd2cf61ce, State: Running, Role: LEADER
I20260812 16:43:10.929121 18560 consensus_queue.cc:237] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } }
I20260812 16:43:10.933530 18069 catalog_manager.cc:5495] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 4edc8a1b568b462f938bee8dd2cf61ce (127.31.61.130). New cstate: current_term: 1 leader_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4edc8a1b568b462f938bee8dd2cf61ce" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 39149 } health_report { overall_health: HEALTHY } } }
I20260812 16:43:10.998056 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:10.990163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.31.61.130:39149)
0812 16:43:10.990393 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:10.990398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:10.990640 (+ 242us) server_negotiation.cc:407] Connection header received
0812 16:43:10.990862 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:10.990868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:10.990977 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:10.991134 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:43:10.991565 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.992773 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.995115 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:10.995782 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:10.995918 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:10.996618 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:43:10.997364 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:43:10.997592 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:10.997678 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 16:43:10.997734 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 16:43:11.008116 18215 raft_consensus.cc:1261] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [term 1 FOLLOWER]: Refusing update from remote peer 4edc8a1b568b462f938bee8dd2cf61ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:11.008170 18488 raft_consensus.cc:1261] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [term 1 FOLLOWER]: Refusing update from remote peer 4edc8a1b568b462f938bee8dd2cf61ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:11.009227 18563 consensus_queue.cc:1035] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef8c4f54f38345e48b78734880eff690" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:11.009735 18560 consensus_queue.cc:1035] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "161dbd09d60147059f4ade5a22f0a26d" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 46449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:11.024441 18566 mvcc.cc:205] Tried to move back new op lower bound from 7317721051157184512 to 7317721050849165312. Current Snapshot: MvccSnapshot[applied={T|T < 7317721051157184512}]
I20260812 16:43:11.045991 18068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:11.046273 18068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:11.049932 18068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1962ba572a324e4f90722eada5bee6a5: Sending DeleteTablet for 3 replicas of tablet 6dc4ae7fec1a407b9fe8e48197859dc2
I20260812 16:43:11.051055 18466 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc4ae7fec1a407b9fe8e48197859dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:11 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:43288
I20260812 16:43:11.051266 18331 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc4ae7fec1a407b9fe8e48197859dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:11 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:60048
I20260812 16:43:11.051539 18195 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc4ae7fec1a407b9fe8e48197859dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:11 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:44632
I20260812 16:43:11.055435 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 18131
I20260812 16:43:11.067452 18575 tablet_replica.cc:330] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: stopping tablet replica
I20260812 16:43:11.067931 18576 tablet_replica.cc:330] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: stopping tablet replica
I20260812 16:43:11.068084 18577 tablet_replica.cc:330] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: stopping tablet replica
I20260812 16:43:11.068540 18577 raft_consensus.cc:2229] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:11.068564 18576 raft_consensus.cc:2229] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:43:11.069096 18577 raft_consensus.cc:2258] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:11.069419 18576 raft_consensus.cc:2258] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:11.069950 18575 raft_consensus.cc:2229] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:11.070451 18575 raft_consensus.cc:2258] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:11.071897 18576 ts_tablet_manager.cc:1905] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:11.073487 18577 ts_tablet_manager.cc:1905] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:11.074280 18575 ts_tablet_manager.cc:1905] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:11.079355 18576 ts_tablet_manager.cc:1918] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:11.080780 18577 ts_tablet_manager.cc:1918] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:11.081064 18577 log.cc:1196] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/6dc4ae7fec1a407b9fe8e48197859dc2
I20260812 16:43:11.081689 18577 ts_tablet_manager.cc:1939] T 6dc4ae7fec1a407b9fe8e48197859dc2 P ef8c4f54f38345e48b78734880eff690: Deleting consensus metadata
Aug 12 16:43:11 dist-test-slave-f7th krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:43:11.083767 18053 catalog_manager.cc:4841] TS ef8c4f54f38345e48b78734880eff690 (127.31.61.129:38227): tablet 6dc4ae7fec1a407b9fe8e48197859dc2 (table test-table [id=75a08a89b02446d99a8effd9501b5804]) successfully deleted
I20260812 16:43:11.087319 18575 ts_tablet_manager.cc:1918] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:11.087531 18575 log.cc:1196] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/6dc4ae7fec1a407b9fe8e48197859dc2
I20260812 16:43:11.080338 18576 log.cc:1196] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/6dc4ae7fec1a407b9fe8e48197859dc2
I20260812 16:43:11.088215 18575 ts_tablet_manager.cc:1939] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 161dbd09d60147059f4ade5a22f0a26d: Deleting consensus metadata
I20260812 16:43:11.088485 18576 ts_tablet_manager.cc:1939] T 6dc4ae7fec1a407b9fe8e48197859dc2 P 4edc8a1b568b462f938bee8dd2cf61ce: Deleting consensus metadata
I20260812 16:43:11.090132 18053 catalog_manager.cc:4841] TS 161dbd09d60147059f4ade5a22f0a26d (127.31.61.131:46449): tablet 6dc4ae7fec1a407b9fe8e48197859dc2 (table test-table [id=75a08a89b02446d99a8effd9501b5804]) successfully deleted
I20260812 16:43:11.090466 18053 catalog_manager.cc:4841] TS 4edc8a1b568b462f938bee8dd2cf61ce (127.31.61.130:39149): tablet 6dc4ae7fec1a407b9fe8e48197859dc2 (table test-table [id=75a08a89b02446d99a8effd9501b5804]) successfully deleted
I20260812 16:43:11.091583 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:11.056931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.31.61.129:38227)
0812 16:43:11.057144 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:11.057152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:11.060919 (+ 3767us) server_negotiation.cc:407] Connection header received
0812 16:43:11.061773 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:11.061780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:11.061910 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:11.061998 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:11.065484 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:11.066829 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:11.070878 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:11.071531 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:11.071655 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:11.083714 (+ 12059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:11.083729 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:11.083749 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:11.083802 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:11.087550 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:11.088680 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:11.088690 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:11.088693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:11.088859 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:11.089326 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:11.089331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:11.089334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:11.089820 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:11.090231 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:11.090976 (+ 745us) server_negotiation.cc:299] Negotiation successful
0812 16:43:11.091202 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:43:11.092226 18256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:11.183977 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 18131
I20260812 16:43:11.206544 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 18267
Aug 12 16:43:11 dist-test-slave-f7th krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:43:11.226100 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:11.207542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.31.61.130:39149)
0812 16:43:11.207739 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:11.207744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:11.207761 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:43:11.207817 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:11.207820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:11.207886 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:11.207967 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:11.209657 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:11.210531 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:11.212481 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:11.213090 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:11.213200 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:11.218681 (+ 5481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:11.218704 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:11.218726 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:11.218784 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:11.222763 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:11.223692 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:11.223704 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:11.223709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:11.223912 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:11.224358 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:11.224371 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:11.224375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:11.224867 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:11.225240 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:11.225464 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 16:43:11.225702 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 16:43:11.226919 18391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:11.341307 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 18267
I20260812 16:43:11.358506 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 18402
Aug 12 16:43:11 dist-test-slave-f7th krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:43:11.374172 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:11.359306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.31.61.131:46449)
0812 16:43:11.359511 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:11.359518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:11.359540 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:43:11.359756 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:11.359760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:11.359827 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:11.359910 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:11.361268 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:11.361966 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:11.363298 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:11.363750 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:11.363866 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:11.368308 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:11.368324 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:11.368341 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:11.368386 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:11.371233 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:11.372079 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:11.372087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:11.372089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:11.372237 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:11.372616 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:11.372620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:11.372622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:11.373002 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:11.373287 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:11.373621 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 16:43:11.373848 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 16:43:11.374823 18528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:11.464236 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 18402
I20260812 16:43:11.478861 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 18037
I20260812 16:43:11.480072 18099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:11.592309 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 18037
2026-08-12T16:43:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4513 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T16:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:11Z Disabled control of system clock
I20260812 16:43:11.668798 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:33533
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:37087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:33533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:43:12.003011 18588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:12.003340 18588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:12.003448 18588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:12.010883 18588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:12.011004 18588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:12.011042 18588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:12.011139 18588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:12.011205 18588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:12.020756 18588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:37087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:33533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:33533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:12.022182 18588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:12.024034 18588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:12.036278 18593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:12.036396 18596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:12.036440 18594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:12.037936 18588 server_base.cc:1034] running on GCE node
I20260812 16:43:12.038774 18588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:12.040266 18588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:12.041494 18588 hybrid_clock.cc:648] HybridClock initialized: now 1786552992041440 us; error 38 us; skew 500 ppm
I20260812 16:43:12.041847 18588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:12.044237 18588 webserver.cc:460] Webserver started at http://127.31.61.190:33025/ using document root <none> and password file <none>
I20260812 16:43:12.044914 18588 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:12.045002 18588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:12.045271 18588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:12.048174 18588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "9b4685be233e40f8bcd5cbc91a4d809f"
format_stamp: "Formatted at 2026-08-12 16:43:12 on dist-test-slave-f7th"
server_key: "a0f765eaba02a73ac928d727dbbb7b59"
server_key_iv: "3a38b0bd7485ab2a410374fe94f7a063"
server_key_version: "encryptionkey@0"
I20260812 16:43:12.048930 18588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "9b4685be233e40f8bcd5cbc91a4d809f"
format_stamp: "Formatted at 2026-08-12 16:43:12 on dist-test-slave-f7th"
server_key: "a0f765eaba02a73ac928d727dbbb7b59"
server_key_iv: "3a38b0bd7485ab2a410374fe94f7a063"
server_key_version: "encryptionkey@0"
I20260812 16:43:12.054399 18588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 16:43:12.058189 18602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:12.059727 18588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:12.059895 18588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "9b4685be233e40f8bcd5cbc91a4d809f"
format_stamp: "Formatted at 2026-08-12 16:43:12 on dist-test-slave-f7th"
server_key: "a0f765eaba02a73ac928d727dbbb7b59"
server_key_iv: "3a38b0bd7485ab2a410374fe94f7a063"
server_key_version: "encryptionkey@0"
I20260812 16:43:12.060108 18588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:12.083946 18588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:12.084856 18588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:12.085160 18588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:12.104537 18588 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:33533
I20260812 16:43:12.104554 18653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:33533 every 8 connection(s)
I20260812 16:43:12.105993 18588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:12.109709 18654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:12.113621 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 18588
I20260812 16:43:12.113793 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:12.114213 31990 external_mini_cluster.cc:1428] Setting key 8add4fc090288d10e302fd0df1915173
I20260812 16:43:12.124420 18654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f: Bootstrap starting.
I20260812 16:43:12.128329 18654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:12.129719 18654 log.cc:824] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:12.130970 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:12.116200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.31.61.190:33533)
0812 16:43:12.117416 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:12.117431 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:12.117486 (+ 55us) server_negotiation.cc:407] Connection header received
0812 16:43:12.119688 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:12.119727 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:12.120434 (+ 707us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:12.120937 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:12.122082 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:12.124008 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:12.125659 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:12.126756 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:12.126933 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:12.128113 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:12.128123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:12.128141 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:12.128936 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:12.129056 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:12.129085 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:12.129205 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:12.129423 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 16:43:12.129784 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":373,"threads_started":1}
I20260812 16:43:12.133037 18654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f: No bootstrap required, opened a new log
I20260812 16:43:12.140478 18654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4685be233e40f8bcd5cbc91a4d809f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33533 } }
I20260812 16:43:12.140944 18654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:12.141047 18654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4685be233e40f8bcd5cbc91a4d809f, State: Initialized, Role: FOLLOWER
I20260812 16:43:12.141667 18654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4685be233e40f8bcd5cbc91a4d809f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33533 } }
I20260812 16:43:12.141913 18654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:12.142050 18654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:12.142198 18654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:12.144483 18654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4685be233e40f8bcd5cbc91a4d809f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33533 } }
I20260812 16:43:12.145031 18654 leader_election.cc:302] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b4685be233e40f8bcd5cbc91a4d809f; no voters:
I20260812 16:43:12.145969 18654 leader_election.cc:288] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:12.145974 18659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:12.146818 18659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [term 1 LEADER]: Becoming Leader. State: Replica: 9b4685be233e40f8bcd5cbc91a4d809f, State: Running, Role: LEADER
I20260812 16:43:12.147400 18659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4685be233e40f8bcd5cbc91a4d809f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33533 } }
I20260812 16:43:12.148751 18654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:12.150347 18661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b4685be233e40f8bcd5cbc91a4d809f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4685be233e40f8bcd5cbc91a4d809f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33533 } } }
I20260812 16:43:12.150830 18661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:12.151315 18660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b4685be233e40f8bcd5cbc91a4d809f. Latest consensus state: current_term: 1 leader_uuid: "9b4685be233e40f8bcd5cbc91a4d809f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4685be233e40f8bcd5cbc91a4d809f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33533 } } }
I20260812 16:43:12.151727 18660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:12.153564 18667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:12.159214 18667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:12.169776 18667 catalog_manager.cc:1347] Generated new cluster ID: 8e73ad468d484723a9da5f4c4824686c
I20260812 16:43:12.169911 18667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:12.195796 18667 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:12.197111 18667 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:12.204255 18667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f: Generated new TSK 0
I20260812 16:43:12.205461 18667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:12.219137 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33533
--builtin_ntp_servers=127.31.61.148:37087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:43:12.558763 18678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:12.559067 18678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:12.559202 18678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:12.566614 18678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:12.566733 18678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:12.566876 18678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:43:12.576877 18678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:37087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:12.578249 18678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:12.580109 18678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:12.593020 18683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:12.593088 18686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:12.593053 18684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:12.594017 18678 server_base.cc:1034] running on GCE node
I20260812 16:43:12.594647 18678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:12.595620 18678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:12.596881 18678 hybrid_clock.cc:648] HybridClock initialized: now 1786552992596832 us; error 37 us; skew 500 ppm
I20260812 16:43:12.597230 18678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:12.599771 18678 webserver.cc:460] Webserver started at http://127.31.61.129:38105/ using document root <none> and password file <none>
I20260812 16:43:12.600497 18678 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:12.600603 18678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:12.600925 18678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:12.604043 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "4bcf2187dd114e318d58a3e57f5e8538"
format_stamp: "Formatted at 2026-08-12 16:43:12 on dist-test-slave-f7th"
server_key: "98c45b8ece715ad6cae9e0d80dd6ea5b"
server_key_iv: "71142f4ad70d4f48d90bfdc683dc235f"
server_key_version: "encryptionkey@0"
I20260812 16:43:12.604825 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "4bcf2187dd114e318d58a3e57f5e8538"
format_stamp: "Formatted at 2026-08-12 16:43:12 on dist-test-slave-f7th"
server_key: "98c45b8ece715ad6cae9e0d80dd6ea5b"
server_key_iv: "71142f4ad70d4f48d90bfdc683dc235f"
server_key_version: "encryptionkey@0"
I20260812 16:43:12.610212 18678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:43:12.613831 18692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:12.615258 18678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:43:12.615432 18678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "4bcf2187dd114e318d58a3e57f5e8538"
format_stamp: "Formatted at 2026-08-12 16:43:12 on dist-test-slave-f7th"
server_key: "98c45b8ece715ad6cae9e0d80dd6ea5b"
server_key_iv: "71142f4ad70d4f48d90bfdc683dc235f"
server_key_version: "encryptionkey@0"
I20260812 16:43:12.615736 18678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:12.649338 18678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:12.650267 18678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:12.650564 18678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:12.651664 18678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:12.653244 18678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:12.653349 18678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:12.653468 18678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:12.653550 18678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:12.687071 18678 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:41671
I20260812 16:43:12.687165 18806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:41671 every 8 connection(s)
I20260812 16:43:12.688428 18678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:43:12.689445 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 18678
I20260812 16:43:12.689625 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:43:12.690167 31990 external_mini_cluster.cc:1428] Setting key b2ee71a4e45b70fce0c3caf227fcc071
I20260812 16:43:12.693797 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33533
--builtin_ntp_servers=127.31.61.148:37087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:43:12.701869 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:12.691239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:38821 (local address 127.31.61.190:33533)
0812 16:43:12.691741 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:12.691747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:12.693119 (+ 1372us) server_negotiation.cc:407] Connection header received
0812 16:43:12.695132 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:12.695137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:12.695225 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:12.695325 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:12.696496 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:12.697238 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:12.698949 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:12.699363 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:12.699481 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:12.700956 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:12.700962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:12.700965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:12.700988 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:12.701043 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:12.701059 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:12.701167 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:12.701402 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 16:43:12.701511 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":230,"threads_started":1}
I20260812 16:43:12.702886 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:12.692003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:33533 (local address 127.31.61.129:38821)
0812 16:43:12.692961 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:12.693007 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:12.694848 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:12.695579 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:12.695588 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:12.695728 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:12.696326 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:12.696344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:12.697386 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:12.697390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:12.698781 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:12.698800 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:12.699638 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:12.699646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:12.699886 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:12.700598 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:43:12.700601 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:12.700644 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:43:12.700731 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:12.701190 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:12.701201 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:12.701208 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 16:43:12.701360 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 16:43:12.701684 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":314,"threads_started":1}
I20260812 16:43:12.704528 18807 heartbeater.cc:347] Connected to a master server at 127.31.61.190:33533
I20260812 16:43:12.704835 18807 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:12.705597 18807 heartbeater.cc:511] Master 127.31.61.190:33533 requested a full tablet report, sending...
I20260812 16:43:12.707938 18619 ts_manager.cc:194] Registered new tserver with Master: 4bcf2187dd114e318d58a3e57f5e8538 (127.31.61.129:41671)
I20260812 16:43:12.709384 18619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.129:38821
W20260812 16:43:13.023972 18811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:13.024290 18811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:13.024401 18811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:13.033001 18811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:13.033128 18811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:13.033280 18811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:43:13.042641 18811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:37087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:13.044039 18811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:13.045847 18811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:13.059166 18817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:13.059223 18819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:13.059245 18816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:13.059924 18811 server_base.cc:1034] running on GCE node
I20260812 16:43:13.060568 18811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:13.061555 18811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:13.062800 18811 hybrid_clock.cc:648] HybridClock initialized: now 1786552993062743 us; error 33 us; skew 500 ppm
I20260812 16:43:13.063164 18811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:13.065663 18811 webserver.cc:460] Webserver started at http://127.31.61.130:46429/ using document root <none> and password file <none>
I20260812 16:43:13.066350 18811 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:13.066459 18811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:13.066749 18811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:13.069824 18811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "2883b8040c994aab9ab793e9946eacd1"
format_stamp: "Formatted at 2026-08-12 16:43:13 on dist-test-slave-f7th"
server_key: "43224ea915276b013e6a29f570e7e7c0"
server_key_iv: "f1ab6605ca507d1344059692fff17bb8"
server_key_version: "encryptionkey@0"
I20260812 16:43:13.070693 18811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "2883b8040c994aab9ab793e9946eacd1"
format_stamp: "Formatted at 2026-08-12 16:43:13 on dist-test-slave-f7th"
server_key: "43224ea915276b013e6a29f570e7e7c0"
server_key_iv: "f1ab6605ca507d1344059692fff17bb8"
server_key_version: "encryptionkey@0"
I20260812 16:43:13.076212 18811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:43:13.079994 18825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:13.081487 18811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:43:13.081641 18811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "2883b8040c994aab9ab793e9946eacd1"
format_stamp: "Formatted at 2026-08-12 16:43:13 on dist-test-slave-f7th"
server_key: "43224ea915276b013e6a29f570e7e7c0"
server_key_iv: "f1ab6605ca507d1344059692fff17bb8"
server_key_version: "encryptionkey@0"
I20260812 16:43:13.081835 18811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:13.092929 18811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:13.093722 18811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:13.093976 18811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:13.094925 18811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:13.096467 18811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:13.096555 18811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:13.096643 18811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:13.096696 18811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:13.129648 18811 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:44519
I20260812 16:43:13.129809 18939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:44519 every 8 connection(s)
I20260812 16:43:13.131095 18811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:43:13.140771 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 18811
I20260812 16:43:13.141026 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:43:13.141837 31990 external_mini_cluster.cc:1428] Setting key 690864833f0d412b144003df5acdcdea
I20260812 16:43:13.144204 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.133886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:46719 (local address 127.31.61.190:33533)
0812 16:43:13.134099 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.134105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.135497 (+ 1392us) server_negotiation.cc:407] Connection header received
0812 16:43:13.137300 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.137307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.137379 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.137467 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:13.138523 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.139249 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.141243 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.141862 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.142011 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.143375 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:13.143380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:13.143383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:13.143400 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:13.143431 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:13.143443 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:13.143550 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.143802 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.143903 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:43:13.144748 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33533
--builtin_ntp_servers=127.31.61.148:37087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:43:13.145320 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.134413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:33533 (local address 127.31.61.130:46719)
0812 16:43:13.135339 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:13.135376 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:13.137034 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:13.137657 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:13.137665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:13.137755 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:13.138347 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.138366 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.139425 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.139429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.140906 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.140920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.142140 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.142146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.142381 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.143059 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:43:13.143062 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:13.143110 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:43:13.143190 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:13.143565 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:13.143577 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:13.143584 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 16:43:13.143788 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 16:43:13.144212 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":286,"threads_started":1}
I20260812 16:43:13.146816 18940 heartbeater.cc:347] Connected to a master server at 127.31.61.190:33533
I20260812 16:43:13.147140 18940 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:13.147945 18940 heartbeater.cc:511] Master 127.31.61.190:33533 requested a full tablet report, sending...
I20260812 16:43:13.149621 18619 ts_manager.cc:194] Registered new tserver with Master: 2883b8040c994aab9ab793e9946eacd1 (127.31.61.130:44519)
I20260812 16:43:13.150662 18619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.130:46719
W20260812 16:43:13.480595 18943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:13.480885 18943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:13.481007 18943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:13.488595 18943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:13.488709 18943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:13.488858 18943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:43:13.498108 18943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:37087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:13.499430 18943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:13.501304 18943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:13.514192 18948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:13.514149 18949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:13.514256 18951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:13.515120 18943 server_base.cc:1034] running on GCE node
I20260812 16:43:13.515811 18943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:13.516804 18943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:13.518010 18943 hybrid_clock.cc:648] HybridClock initialized: now 1786552993517966 us; error 47 us; skew 500 ppm
I20260812 16:43:13.518323 18943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:13.520704 18943 webserver.cc:460] Webserver started at http://127.31.61.131:40557/ using document root <none> and password file <none>
I20260812 16:43:13.521421 18943 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:13.521510 18943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:13.521785 18943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:13.524734 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "f1e5061ea7624c58b87b0b546c45d84a"
format_stamp: "Formatted at 2026-08-12 16:43:13 on dist-test-slave-f7th"
server_key: "815470bbfca0ed61e4e651b83d415add"
server_key_iv: "881c3a855c5aed24ce6abdee1ce68e5a"
server_key_version: "encryptionkey@0"
I20260812 16:43:13.525476 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "f1e5061ea7624c58b87b0b546c45d84a"
format_stamp: "Formatted at 2026-08-12 16:43:13 on dist-test-slave-f7th"
server_key: "815470bbfca0ed61e4e651b83d415add"
server_key_iv: "881c3a855c5aed24ce6abdee1ce68e5a"
server_key_version: "encryptionkey@0"
I20260812 16:43:13.530640 18943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:43:13.534309 18957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:13.535872 18943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:13.536032 18943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "f1e5061ea7624c58b87b0b546c45d84a"
format_stamp: "Formatted at 2026-08-12 16:43:13 on dist-test-slave-f7th"
server_key: "815470bbfca0ed61e4e651b83d415add"
server_key_iv: "881c3a855c5aed24ce6abdee1ce68e5a"
server_key_version: "encryptionkey@0"
I20260812 16:43:13.536238 18943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:13.567564 18943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:13.568501 18943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:13.568795 18943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:13.569851 18943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:13.571391 18943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:13.571483 18943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:13.571587 18943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:13.571677 18943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 16:43:13.604539 18943 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:37349
I20260812 16:43:13.604635 19070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:37349 every 8 connection(s)
I20260812 16:43:13.605810 18943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:43:13.612835 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 18943
I20260812 16:43:13.613031 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:43:13.613432 31990 external_mini_cluster.cc:1428] Setting key ab7e5a91d68ac74bcecc7b92176b70f7
I20260812 16:43:13.619666 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.608473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:50271 (local address 127.31.61.190:33533)
0812 16:43:13.608695 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.608702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.610038 (+ 1336us) server_negotiation.cc:407] Connection header received
0812 16:43:13.611804 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.611810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.611894 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.611981 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:13.613756 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.614778 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.616434 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.616893 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.617013 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.618268 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:13.618274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:13.618277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:13.618295 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:13.618328 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:13.618351 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:13.618425 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.618987 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.619211 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 16:43:13.620848 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.608947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:33533 (local address 127.31.61.131:50271)
0812 16:43:13.609855 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:13.609918 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:13.611508 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:13.612456 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:13.612465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:13.612696 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:13.613446 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.613467 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.614932 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.614936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.616290 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.616303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.617121 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.617129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.617345 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.617948 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:43:13.617950 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:13.618005 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:43:13.618084 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:13.618440 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:13.618451 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:13.618459 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 16:43:13.618943 (+ 484us) client_negotiation.cc:240] Negotiation successful
0812 16:43:13.619576 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":274,"threads_started":1}
I20260812 16:43:13.622277 19071 heartbeater.cc:347] Connected to a master server at 127.31.61.190:33533
I20260812 16:43:13.622578 19071 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:13.623317 19071 heartbeater.cc:511] Master 127.31.61.190:33533 requested a full tablet report, sending...
I20260812 16:43:13.625058 18619 ts_manager.cc:194] Registered new tserver with Master: f1e5061ea7624c58b87b0b546c45d84a (127.31.61.131:37349)
I20260812 16:43:13.625861 18619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.131:50271
I20260812 16:43:13.628535 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:43:13.639169 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.632785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.31.61.190:33533)
0812 16:43:13.633036 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.633040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.633467 (+ 427us) server_negotiation.cc:407] Connection header received
0812 16:43:13.633581 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.633585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.633663 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.633755 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:13.634435 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.635464 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.637015 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.637437 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.637552 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.638557 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:13.638582 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:13.638585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:13.638604 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:13.638636 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:13.638648 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:13.638764 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.638883 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.638934 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 16:43:13.643311 18619 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:43:13.650174 18619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:43:13.688917 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.663121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:41671 (local address 127.0.0.1:60622)
0812 16:43:13.679319 (+ 16198us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:13.679342 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:13.679534 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:13.680502 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:13.680510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:13.680543 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:13.681115 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.681129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.683096 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.683101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.685479 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.685496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.687854 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.687861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.688102 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.688158 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 16:43:13.688290 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 16:43:13.688504 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16057,"mutex_wait_us":193,"spinlock_wait_cycles":5376,"thread_start_us":15876,"threads_started":1}
I20260812 16:43:13.689888 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.663268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.31.61.129:41671)
0812 16:43:13.677053 (+ 13785us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.677063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.679712 (+ 2649us) server_negotiation.cc:407] Connection header received
0812 16:43:13.679832 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.679841 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.680217 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.680498 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:13.681357 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.682970 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.685678 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.687084 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.687451 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.687719 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.689310 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.689516 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13591,"thread_start_us":13469,"threads_started":1}
I20260812 16:43:13.689961 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.665715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.31.61.130:44519)
0812 16:43:13.678832 (+ 13117us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.678842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.679842 (+ 1000us) server_negotiation.cc:407] Connection header received
0812 16:43:13.680966 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.680975 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.681204 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.681386 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:13.682064 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.683418 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.686071 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.688034 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.688172 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.688396 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.689463 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.689622 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12939,"thread_start_us":12817,"threads_started":1}
I20260812 16:43:13.689975 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.663183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.31.61.131:37349)
0812 16:43:13.680417 (+ 17234us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.680424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.680456 (+ 32us) server_negotiation.cc:407] Connection header received
0812 16:43:13.680543 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.680550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.680761 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.681648 (+ 887us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:13.681908 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.682923 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.686927 (+ 4004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.688308 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.688431 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.688698 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.689442 (+ 744us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.689620 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17033,"thread_start_us":16876,"threads_started":1}
I20260812 16:43:13.689347 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.665187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:44519 (local address 127.0.0.1:44384)
0812 16:43:13.679697 (+ 14510us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:13.679719 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:13.680685 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:13.681483 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:13.681486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:13.681513 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:13.681883 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.681895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.683618 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.683624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.685803 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.685818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.688529 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.688535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.688708 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.688743 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 16:43:13.688830 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 16:43:13.688965 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14352,"mutex_wait_us":4,"thread_start_us":12178,"threads_started":1}
I20260812 16:43:13.690380 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.662835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:37349 (local address 127.0.0.1:45434)
0812 16:43:13.677883 (+ 15048us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:13.677950 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:13.678198 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:13.680985 (+ 2787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:13.680990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:13.681014 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:13.681467 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.681479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.683053 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.683058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.686713 (+ 3655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.686728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.688824 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.688830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.688991 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.689019 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 16:43:13.689093 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:43:13.689209 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14693,"mutex_wait_us":1349,"thread_start_us":16526,"threads_started":1}
I20260812 16:43:13.693887 18874 tablet_service.cc:1467] Processing CreateTablet for tablet aa1b730c4cd94fa78bc20909a3e2f658 (DEFAULT_TABLE table=test-table [id=f8d56e25c5c64c67aa22f1cf84b60d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:13.694181 18742 tablet_service.cc:1467] Processing CreateTablet for tablet aa1b730c4cd94fa78bc20909a3e2f658 (DEFAULT_TABLE table=test-table [id=f8d56e25c5c64c67aa22f1cf84b60d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:13.695231 18874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1b730c4cd94fa78bc20909a3e2f658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:13.695395 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1b730c4cd94fa78bc20909a3e2f658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:13.697160 19005 tablet_service.cc:1467] Processing CreateTablet for tablet aa1b730c4cd94fa78bc20909a3e2f658 (DEFAULT_TABLE table=test-table [id=f8d56e25c5c64c67aa22f1cf84b60d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:13.698464 19005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1b730c4cd94fa78bc20909a3e2f658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:13.707219 19089 tablet_bootstrap.cc:492] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: Bootstrap starting.
I20260812 16:43:13.709992 19090 tablet_bootstrap.cc:492] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: Bootstrap starting.
I20260812 16:43:13.712425 18807 heartbeater.cc:503] Master 127.31.61.190:33533 was elected leader, sending a full tablet report...
I20260812 16:43:13.712906 19089 tablet_bootstrap.cc:654] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:13.713743 19090 tablet_bootstrap.cc:654] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:13.714283 19089 log.cc:824] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:13.714931 19090 log.cc:824] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:13.718214 19091 tablet_bootstrap.cc:492] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: Bootstrap starting.
I20260812 16:43:13.721737 19090 tablet_bootstrap.cc:492] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: No bootstrap required, opened a new log
I20260812 16:43:13.721817 19089 tablet_bootstrap.cc:492] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: No bootstrap required, opened a new log
I20260812 16:43:13.722016 19090 ts_tablet_manager.cc:1397] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260812 16:43:13.722127 19091 tablet_bootstrap.cc:654] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:13.722148 19089 ts_tablet_manager.cc:1397] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260812 16:43:13.723353 19091 log.cc:824] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:13.726408 19091 tablet_bootstrap.cc:492] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: No bootstrap required, opened a new log
I20260812 16:43:13.726720 19091 ts_tablet_manager.cc:1397] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 16:43:13.730273 19090 raft_consensus.cc:348] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } }
I20260812 16:43:13.730679 19090 raft_consensus.cc:374] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:13.730801 19090 raft_consensus.cc:729] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2883b8040c994aab9ab793e9946eacd1, State: Initialized, Role: FOLLOWER
I20260812 16:43:13.731483 19090 consensus_queue.cc:260] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } }
I20260812 16:43:13.731868 19089 raft_consensus.cc:348] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } }
I20260812 16:43:13.732450 19089 raft_consensus.cc:374] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:13.732666 19089 raft_consensus.cc:729] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcf2187dd114e318d58a3e57f5e8538, State: Initialized, Role: FOLLOWER
I20260812 16:43:13.732744 18940 heartbeater.cc:503] Master 127.31.61.190:33533 was elected leader, sending a full tablet report...
I20260812 16:43:13.733371 19090 ts_tablet_manager.cc:1428] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260812 16:43:13.733268 19089 consensus_queue.cc:260] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } }
I20260812 16:43:13.735020 19089 ts_tablet_manager.cc:1428] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260812 16:43:13.738426 19091 raft_consensus.cc:348] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } }
I20260812 16:43:13.738874 19091 raft_consensus.cc:374] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:13.738977 19091 raft_consensus.cc:729] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e5061ea7624c58b87b0b546c45d84a, State: Initialized, Role: FOLLOWER
I20260812 16:43:13.739535 19091 consensus_queue.cc:260] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } }
I20260812 16:43:13.740571 19071 heartbeater.cc:503] Master 127.31.61.190:33533 was elected leader, sending a full tablet report...
I20260812 16:43:13.741071 19091 ts_tablet_manager.cc:1428] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260812 16:43:13.814108 19097 raft_consensus.cc:482] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:43:13.814404 19097 raft_consensus.cc:504] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } }
I20260812 16:43:13.816138 19097 leader_election.cc:288] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcf2187dd114e318d58a3e57f5e8538 (127.31.61.129:41671), 2883b8040c994aab9ab793e9946eacd1 (127.31.61.130:44519)
I20260812 16:43:13.821540 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.816390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:41671 (local address 127.31.61.131:50257)
0812 16:43:13.816755 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:13.816774 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:13.816901 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:13.817274 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:13.817279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:13.817306 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:13.817472 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.817482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.818437 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.818440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.819775 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.819787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.820834 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.820841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.820995 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.821017 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 16:43:13.821113 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 16:43:13.821249 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":1}
I20260812 16:43:13.821538 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.816550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:50257 (local address 127.31.61.129:41671)
0812 16:43:13.816761 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.816766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.816866 (+ 100us) server_negotiation.cc:407] Connection header received
0812 16:43:13.817043 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.817048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.817120 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.817199 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:13.817608 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.818288 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.819913 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.820681 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.820845 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.820926 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.821163 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.821287 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 16:43:13.822477 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.816662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:44519 (local address 127.31.61.131:47637)
0812 16:43:13.817406 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:13.817423 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:13.817531 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:13.817881 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:13.817885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:13.817907 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:13.818205 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.818214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.819179 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.819185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.820718 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:13.820731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.821903 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.821909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:13.822078 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.822112 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:43:13.822182 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 16:43:13.822264 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":414,"threads_started":1}
I20260812 16:43:13.822654 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.816581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:47637 (local address 127.31.61.130:44519)
0812 16:43:13.816755 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.816761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.817509 (+ 748us) server_negotiation.cc:407] Connection header received
0812 16:43:13.817696 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.817700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.817769 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.817841 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:13.818362 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.819031 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.820944 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.821753 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.821896 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.821960 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.822238 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.822370 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 16:43:13.822633 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1b730c4cd94fa78bc20909a3e2f658" candidate_uuid: "f1e5061ea7624c58b87b0b546c45d84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcf2187dd114e318d58a3e57f5e8538" is_pre_election: true
I20260812 16:43:13.823225 18762 raft_consensus.cc:2454] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e5061ea7624c58b87b0b546c45d84a in term 0.
I20260812 16:43:13.823410 18894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1b730c4cd94fa78bc20909a3e2f658" candidate_uuid: "f1e5061ea7624c58b87b0b546c45d84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2883b8040c994aab9ab793e9946eacd1" is_pre_election: true
I20260812 16:43:13.823884 18958 leader_election.cc:302] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcf2187dd114e318d58a3e57f5e8538, f1e5061ea7624c58b87b0b546c45d84a; no voters:
I20260812 16:43:13.823885 18894 raft_consensus.cc:2454] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e5061ea7624c58b87b0b546c45d84a in term 0.
I20260812 16:43:13.824286 19097 raft_consensus.cc:2781] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:43:13.824426 19097 raft_consensus.cc:482] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:43:13.824530 19097 raft_consensus.cc:3037] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:13.826699 19097 raft_consensus.cc:504] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } }
I20260812 16:43:13.827325 19097 leader_election.cc:288] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [CANDIDATE]: Term 1 election: Requested vote from peers 4bcf2187dd114e318d58a3e57f5e8538 (127.31.61.129:41671), 2883b8040c994aab9ab793e9946eacd1 (127.31.61.130:44519)
I20260812 16:43:13.827766 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1b730c4cd94fa78bc20909a3e2f658" candidate_uuid: "f1e5061ea7624c58b87b0b546c45d84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcf2187dd114e318d58a3e57f5e8538"
I20260812 16:43:13.827805 18894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1b730c4cd94fa78bc20909a3e2f658" candidate_uuid: "f1e5061ea7624c58b87b0b546c45d84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2883b8040c994aab9ab793e9946eacd1"
I20260812 16:43:13.827958 18762 raft_consensus.cc:3037] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:13.827967 18894 raft_consensus.cc:3037] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:13.830108 18894 raft_consensus.cc:2454] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e5061ea7624c58b87b0b546c45d84a in term 1.
I20260812 16:43:13.830180 18762 raft_consensus.cc:2454] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e5061ea7624c58b87b0b546c45d84a in term 1.
I20260812 16:43:13.830482 18961 leader_election.cc:302] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2883b8040c994aab9ab793e9946eacd1, f1e5061ea7624c58b87b0b546c45d84a; no voters:
I20260812 16:43:13.830780 19097 raft_consensus.cc:2781] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:13.831314 19097 raft_consensus.cc:686] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 1 LEADER]: Becoming Leader. State: Replica: f1e5061ea7624c58b87b0b546c45d84a, State: Running, Role: LEADER
I20260812 16:43:13.831836 19097 consensus_queue.cc:237] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } }
I20260812 16:43:13.837654 18618 catalog_manager.cc:5495] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e5061ea7624c58b87b0b546c45d84a (127.31.61.131). New cstate: current_term: 1 leader_uuid: "f1e5061ea7624c58b87b0b546c45d84a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e5061ea7624c58b87b0b546c45d84a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 37349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 } health_report { overall_health: UNKNOWN } } }
W20260812 16:43:13.857460 19072 tablet.cc:2369] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:43:13.883345 18941 tablet.cc:2369] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:13.891077 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.885802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.31.61.131:37349)
0812 16:43:13.886014 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.886018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.886074 (+ 56us) server_negotiation.cc:407] Connection header received
0812 16:43:13.886255 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.886259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.886339 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.886467 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:43:13.886857 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.887591 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.888972 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.889388 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.889498 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.889941 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:43:13.890467 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:43:13.890604 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.890777 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.890813 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:43:13.899308 18762 raft_consensus.cc:1261] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [term 1 FOLLOWER]: Refusing update from remote peer f1e5061ea7624c58b87b0b546c45d84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:13.899308 18894 raft_consensus.cc:1261] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [term 1 FOLLOWER]: Refusing update from remote peer f1e5061ea7624c58b87b0b546c45d84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:13.900358 19099 consensus_queue.cc:1035] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2883b8040c994aab9ab793e9946eacd1" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 44519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:13.900776 19097 consensus_queue.cc:1035] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcf2187dd114e318d58a3e57f5e8538" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 41671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:13.913635 19105 mvcc.cc:205] Tried to move back new op lower bound from 7317721063001276416 to 7317721062742044672. Current Snapshot: MvccSnapshot[applied={T|T < 7317721063001276416}]
I20260812 16:43:13.915565 19104 mvcc.cc:205] Tried to move back new op lower bound from 7317721063001276416 to 7317721062742044672. Current Snapshot: MvccSnapshot[applied={T|T < 7317721063001276416}]
I20260812 16:43:13.925401 18618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:13.925662 18618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:13.929313 18618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b4685be233e40f8bcd5cbc91a4d809f: Sending DeleteTablet for 3 replicas of tablet aa1b730c4cd94fa78bc20909a3e2f658
I20260812 16:43:13.930536 18874 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1b730c4cd94fa78bc20909a3e2f658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:13 UTC) from {username='slave'} at 127.0.0.1:44384
I20260812 16:43:13.930547 19005 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1b730c4cd94fa78bc20909a3e2f658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:13 UTC) from {username='slave'} at 127.0.0.1:45434
I20260812 16:43:13.930631 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1b730c4cd94fa78bc20909a3e2f658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:13 UTC) from {username='slave'} at 127.0.0.1:60622
I20260812 16:43:13.933461 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 18678
I20260812 16:43:13.941500 19115 tablet_replica.cc:330] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: stopping tablet replica
I20260812 16:43:13.942350 19115 raft_consensus.cc:2229] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:13.942889 19115 raft_consensus.cc:2258] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:13.943060 19113 tablet_replica.cc:330] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: stopping tablet replica
I20260812 16:43:13.944358 19114 tablet_replica.cc:330] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: stopping tablet replica
I20260812 16:43:13.944804 19114 raft_consensus.cc:2229] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:13.944996 19115 ts_tablet_manager.cc:1905] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:13.945307 19114 raft_consensus.cc:2258] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:13.945993 19113 raft_consensus.cc:2229] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:43:13.946713 19113 raft_consensus.cc:2258] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:13.949506 19114 ts_tablet_manager.cc:1905] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:13.951251 19113 ts_tablet_manager.cc:1905] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:13.952365 19115 ts_tablet_manager.cc:1918] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:13.952582 19115 log.cc:1196] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/aa1b730c4cd94fa78bc20909a3e2f658
I20260812 16:43:13.953220 19115 ts_tablet_manager.cc:1939] T aa1b730c4cd94fa78bc20909a3e2f658 P 4bcf2187dd114e318d58a3e57f5e8538: Deleting consensus metadata
I20260812 16:43:13.955147 18603 catalog_manager.cc:4841] TS 4bcf2187dd114e318d58a3e57f5e8538 (127.31.61.129:41671): tablet aa1b730c4cd94fa78bc20909a3e2f658 (table test-table [id=f8d56e25c5c64c67aa22f1cf84b60d0f]) successfully deleted
I20260812 16:43:13.958209 19113 ts_tablet_manager.cc:1918] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:13.958400 19113 log.cc:1196] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/aa1b730c4cd94fa78bc20909a3e2f658
I20260812 16:43:13.958604 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:13.934681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.31.61.129:41671)
0812 16:43:13.934868 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:13.934875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:13.950495 (+ 15620us) server_negotiation.cc:407] Connection header received
0812 16:43:13.951007 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:13.951014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:13.951114 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:13.951193 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:13.951991 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.953143 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.955134 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:13.955748 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:13.955872 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:13.957054 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:13.957064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:13.957081 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:13.957781 (+ 700us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:13.957846 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:13.957874 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:13.958021 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:13.958141 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 16:43:13.958257 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 16:43:13.958947 19113 ts_tablet_manager.cc:1939] T aa1b730c4cd94fa78bc20909a3e2f658 P f1e5061ea7624c58b87b0b546c45d84a: Deleting consensus metadata
I20260812 16:43:13.959543 18802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:13.965448 19114 ts_tablet_manager.cc:1918] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:13.965602 19114 log.cc:1196] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/aa1b730c4cd94fa78bc20909a3e2f658
I20260812 16:43:13.966030 19114 ts_tablet_manager.cc:1939] T aa1b730c4cd94fa78bc20909a3e2f658 P 2883b8040c994aab9ab793e9946eacd1: Deleting consensus metadata
I20260812 16:43:13.967664 18606 catalog_manager.cc:4841] TS 2883b8040c994aab9ab793e9946eacd1 (127.31.61.130:44519): tablet aa1b730c4cd94fa78bc20909a3e2f658 (table test-table [id=f8d56e25c5c64c67aa22f1cf84b60d0f]) successfully deleted
I20260812 16:43:13.968493 18605 catalog_manager.cc:4841] TS f1e5061ea7624c58b87b0b546c45d84a (127.31.61.131:37349): tablet aa1b730c4cd94fa78bc20909a3e2f658 (table test-table [id=f8d56e25c5c64c67aa22f1cf84b60d0f]) successfully deleted
I20260812 16:43:14.072620 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 18678
I20260812 16:43:14.088943 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 18811
I20260812 16:43:14.096432 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:14.089726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.31.61.130:44519)
0812 16:43:14.089921 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:14.089927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:14.089950 (+ 23us) server_negotiation.cc:407] Connection header received
0812 16:43:14.090021 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:14.090027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:14.090138 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:14.090261 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:14.090605 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:14.091779 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:14.093048 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:14.093697 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:14.093823 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:14.094806 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:14.094817 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:14.094839 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:14.095655 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:14.095718 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:14.095744 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:14.095859 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:14.095978 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 16:43:14.096093 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 16:43:14.097188 18935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:14.199030 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 18811
I20260812 16:43:14.215281 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 18943
I20260812 16:43:14.222712 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:14.216046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.31.61.131:37349)
0812 16:43:14.216240 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:14.216246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:14.216365 (+ 119us) server_negotiation.cc:407] Connection header received
0812 16:43:14.216536 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:14.216541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:14.216647 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:14.216732 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:14.217270 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:14.218213 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:14.219494 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:14.220017 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:14.220132 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:14.221277 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:14.221286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:14.221302 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:14.221935 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:14.221986 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:14.222007 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:14.222114 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:14.222248 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 16:43:14.222361 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 16:43:14.223433 19066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:14.346487 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 18943
I20260812 16:43:14.364130 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 18588
I20260812 16:43:14.365376 18649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:14.481406 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 18588
2026-08-12T16:43:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2888 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:14 dist-test-slave-f7th krb5kdc[19123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:14 dist-test-slave-f7th krb5kdc[19123](info): set up 2 sockets
Aug 12 16:43:14 dist-test-slave-f7th krb5kdc[19123](info): commencing operation
krb5kdc: starting...
W20260812 16:43:15.536960 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:15 dist-test-slave-f7th krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:15.812179 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:41577
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:16.151095 19136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:16.151420 19136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:16.151544 19136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:16.159509 19136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:16.159700 19136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:16.159792 19136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:16.159868 19136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:16.159932 19136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:16.170248 19136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:41577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:16.171715 19136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:16.173074 19136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:16.186421 19142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:16.186630 19141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:16.187103 19136 server_base.cc:1034] running on GCE node
W20260812 16:43:16.187216 19145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:16.188127 19136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:16.189790 19136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:16.191039 19136 hybrid_clock.cc:648] HybridClock initialized: now 1786552996190990 us; error 35 us; skew 500 ppm
I20260812 16:43:16.191457 19136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:16 dist-test-slave-f7th krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552996, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:16.199947 19136 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:16.201907 19136 webserver.cc:460] Webserver started at http://127.31.61.190:35813/ using document root <none> and password file <none>
I20260812 16:43:16.202665 19136 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:16.202781 19136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:16.203107 19136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:16.206358 19136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "fc113fc59d254b83a0bb493cd941428d"
format_stamp: "Formatted at 2026-08-12 16:43:16 on dist-test-slave-f7th"
server_key: "f8a2acbd63134fd181b120ae5d3deb92"
server_key_iv: "0ce60bcfa64fe5f66c8c9e35aa90740b"
server_key_version: "encryptionkey@0"
I20260812 16:43:16.207183 19136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "fc113fc59d254b83a0bb493cd941428d"
format_stamp: "Formatted at 2026-08-12 16:43:16 on dist-test-slave-f7th"
server_key: "f8a2acbd63134fd181b120ae5d3deb92"
server_key_iv: "0ce60bcfa64fe5f66c8c9e35aa90740b"
server_key_version: "encryptionkey@0"
I20260812 16:43:16.213068 19136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 16:43:16.217029 19152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:16.218493 19136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 16:43:16.218667 19136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "fc113fc59d254b83a0bb493cd941428d"
format_stamp: "Formatted at 2026-08-12 16:43:16 on dist-test-slave-f7th"
server_key: "f8a2acbd63134fd181b120ae5d3deb92"
server_key_iv: "0ce60bcfa64fe5f66c8c9e35aa90740b"
server_key_version: "encryptionkey@0"
I20260812 16:43:16.218863 19136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:16.243852 19136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:16.248500 19136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:16.248807 19136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:16.267887 19136 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:41577
I20260812 16:43:16.267921 19203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:41577 every 8 connection(s)
I20260812 16:43:16.269295 19136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:16.273293 19204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:16.279580 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 19136
I20260812 16:43:16.279810 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:16.280499 31990 external_mini_cluster.cc:1428] Setting key d2888697493965fbab9b0a847717c1b8
I20260812 16:43:16.285517 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d: Bootstrap starting.
I20260812 16:43:16.289271 19204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:16.290547 19204 log.cc:824] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:16.294073 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d: No bootstrap required, opened a new log
Aug 12 16:43:16 dist-test-slave-f7th krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:16.303388 19204 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc113fc59d254b83a0bb493cd941428d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41577 } }
I20260812 16:43:16.303954 19204 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:16.304073 19204 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc113fc59d254b83a0bb493cd941428d, State: Initialized, Role: FOLLOWER
I20260812 16:43:16.304879 19204 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc113fc59d254b83a0bb493cd941428d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41577 } }
I20260812 16:43:16.305191 19204 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:16.305368 19204 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:16.305560 19204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:16.307068 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:16.282830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.31.61.190:41577)
0812 16:43:16.283732 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:16.283746 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:16.283789 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:43:16.285720 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:16.285746 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:16.286158 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:16.286500 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:16.288385 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:16.289799 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:16.291625 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:16.292475 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:16.292624 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:16.297681 (+ 5057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:16.297689 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:16.297709 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:16.297748 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:16.300885 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:16.301854 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:16.301866 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:16.301874 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:16.302073 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:16.302537 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:16.302542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:16.302545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:16.303422 (+ 877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:16.303822 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:16.305104 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0812 16:43:16.305617 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":398,"threads_started":1}
I20260812 16:43:16.308741 19204 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc113fc59d254b83a0bb493cd941428d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41577 } }
I20260812 16:43:16.309401 19204 leader_election.cc:302] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc113fc59d254b83a0bb493cd941428d; no voters:
I20260812 16:43:16.310213 19204 leader_election.cc:288] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:16.310400 19209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:16.311755 19209 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [term 1 LEADER]: Becoming Leader. State: Replica: fc113fc59d254b83a0bb493cd941428d, State: Running, Role: LEADER
I20260812 16:43:16.312439 19209 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc113fc59d254b83a0bb493cd941428d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41577 } }
I20260812 16:43:16.317515 19204 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:16.325512 19210 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc113fc59d254b83a0bb493cd941428d. Latest consensus state: current_term: 1 leader_uuid: "fc113fc59d254b83a0bb493cd941428d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc113fc59d254b83a0bb493cd941428d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41577 } } }
I20260812 16:43:16.326232 19210 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:16.327764 19211 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc113fc59d254b83a0bb493cd941428d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc113fc59d254b83a0bb493cd941428d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41577 } } }
I20260812 16:43:16.328014 19211 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:16.328651 19221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:16.334373 19221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:16.343365 19221 catalog_manager.cc:1347] Generated new cluster ID: 4e4cae27047c4e53974f58f2fc40f973
I20260812 16:43:16.343508 19221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:16.360967 19221 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:16.362920 19221 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:16.374024 19221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d: Generated new TSK 0
I20260812 16:43:16.374889 19221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.61.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
Entry for principal kudu/127.31.61.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab.
I20260812 16:43:16.444159 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41577
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:16.777498 19231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:16.777781 19231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:16.777874 19231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:16.785490 19231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:16.785615 19231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:16.785732 19231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:43:16.795809 19231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.129.keytab
--principal=kudu/127.31.61.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:16.797089 19231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:16.798355 19231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:16.812098 19237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:16.812125 19238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:16.812049 19240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:16.812693 19231 server_base.cc:1034] running on GCE node
I20260812 16:43:16.813237 19231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:16.814196 19231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:16.815419 19231 hybrid_clock.cc:648] HybridClock initialized: now 1786552996815371 us; error 36 us; skew 500 ppm
I20260812 16:43:16.815819 19231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:16 dist-test-slave-f7th krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552996, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:16.819119 19231 init.cc:377] Logged in from keytab as kudu/127.31.61.129@KRBTEST.COM (short username kudu)
I20260812 16:43:16.820593 19231 webserver.cc:460] Webserver started at http://127.31.61.129:42837/ using document root <none> and password file <none>
I20260812 16:43:16.821266 19231 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:16.821367 19231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:16.821704 19231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:16.824841 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "47841c2b4c634114bcf53564a72619ab"
format_stamp: "Formatted at 2026-08-12 16:43:16 on dist-test-slave-f7th"
server_key: "bf52d9c3d690a4db452b4c8cb49cae03"
server_key_iv: "c943e43506ac146405ea1a91c5d8f4ce"
server_key_version: "encryptionkey@0"
I20260812 16:43:16.825613 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "47841c2b4c634114bcf53564a72619ab"
format_stamp: "Formatted at 2026-08-12 16:43:16 on dist-test-slave-f7th"
server_key: "bf52d9c3d690a4db452b4c8cb49cae03"
server_key_iv: "c943e43506ac146405ea1a91c5d8f4ce"
server_key_version: "encryptionkey@0"
I20260812 16:43:16.830895 19231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 16:43:16.834574 19247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:16.835944 19231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:16.836081 19231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "47841c2b4c634114bcf53564a72619ab"
format_stamp: "Formatted at 2026-08-12 16:43:16 on dist-test-slave-f7th"
server_key: "bf52d9c3d690a4db452b4c8cb49cae03"
server_key_iv: "c943e43506ac146405ea1a91c5d8f4ce"
server_key_version: "encryptionkey@0"
I20260812 16:43:16.836259 19231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:16.850543 19231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:16.854735 19231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:16.854991 19231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:16.855991 19231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:16.857306 19231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:16.857378 19231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:16.857460 19231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:16.857497 19231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:16.887418 19231 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:42181
I20260812 16:43:16.887446 19359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:42181 every 8 connection(s)
I20260812 16:43:16.888649 19231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:43:16.890414 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 19231
I20260812 16:43:16.890573 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:43:16.891129 31990 external_mini_cluster.cc:1428] Setting key 9578f3e9fcba8ef16f0166a69eb68429
Aug 12 16:43:16 dist-test-slave-f7th krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552996, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.129@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:16.910789 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:16.892182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:43257 (local address 127.31.61.190:41577)
0812 16:43:16.895937 (+ 3755us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:16.895942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:16.895956 (+ 14us) server_negotiation.cc:407] Connection header received
0812 16:43:16.896004 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:16.896007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:16.896061 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:16.896141 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:16.898049 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:16.898818 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:16.900918 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:16.901341 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:16.901474 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:16.904939 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:16.904946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:16.904951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:16.904992 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:16.907826 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:16.908546 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:16.908551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:16.908555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:16.908696 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:16.909106 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:16.909110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:16.909113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:16.909427 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:16.909565 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:16.910216 (+ 651us) server_negotiation.cc:299] Negotiation successful
0812 16:43:16.910436 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3662,"thread_start_us":3562,"threads_started":1}
I20260812 16:43:16.911557 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:16.891692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:41577 (local address 127.31.61.129:43257)
0812 16:43:16.892590 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:16.892656 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:16.895117 (+ 2461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:16.896415 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:16.896426 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:16.897104 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:16.897848 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:16.897871 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:16.899002 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:16.899008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:16.900759 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:16.900775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:16.901600 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:16.901606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:16.901860 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:16.902565 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:16.902600 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:16.904707 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:16.907966 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:16.907973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:16.907986 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:16.908412 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:16.908808 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:16.908811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:16.908814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:16.909003 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:16.909532 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:16.909537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:16.909853 (+ 316us) client_negotiation.cc:769] Sending connection context
0812 16:43:16.910069 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 16:43:16.910472 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":326,"threads_started":1}
I20260812 16:43:16.913079 19360 heartbeater.cc:347] Connected to a master server at 127.31.61.190:41577
I20260812 16:43:16.913360 19360 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:16.914108 19360 heartbeater.cc:511] Master 127.31.61.190:41577 requested a full tablet report, sending...
I20260812 16:43:16.916239 19169 ts_manager.cc:194] Registered new tserver with Master: 47841c2b4c634114bcf53564a72619ab (127.31.61.129:42181)
I20260812 16:43:16.917699 19169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.129@KRBTEST.COM'} at 127.31.61.129:43257
WARNING: no policy specified for kudu/127.31.61.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
Entry for principal kudu/127.31.61.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab.
I20260812 16:43:16.951632 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41577
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:17.282578 19366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:17.282876 19366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:17.283002 19366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:17.290781 19366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:17.290917 19366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:17.291069 19366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:43:17.301148 19366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.130.keytab
--principal=kudu/127.31.61.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:17.302697 19366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:17.304179 19366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:17.318221 19373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:17.318252 19372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:17.318713 19375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:17.321105 19366 server_base.cc:1034] running on GCE node
I20260812 16:43:17.321810 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:17.322888 19366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:17.324107 19366 hybrid_clock.cc:648] HybridClock initialized: now 1786552997324037 us; error 56 us; skew 500 ppm
I20260812 16:43:17.324443 19366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:17 dist-test-slave-f7th krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552997, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:17.327638 19366 init.cc:377] Logged in from keytab as kudu/127.31.61.130@KRBTEST.COM (short username kudu)
I20260812 16:43:17.329218 19366 webserver.cc:460] Webserver started at http://127.31.61.130:36875/ using document root <none> and password file <none>
I20260812 16:43:17.329944 19366 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:17.330065 19366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:17.330366 19366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:17.333391 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "e085854173a44ca9b4b72f6c17962ed2"
format_stamp: "Formatted at 2026-08-12 16:43:17 on dist-test-slave-f7th"
server_key: "9a6ee2328d55a6cd283d1181c00d71a5"
server_key_iv: "c6124e18947cdf7938ac327edbd2941d"
server_key_version: "encryptionkey@0"
I20260812 16:43:17.334205 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "e085854173a44ca9b4b72f6c17962ed2"
format_stamp: "Formatted at 2026-08-12 16:43:17 on dist-test-slave-f7th"
server_key: "9a6ee2328d55a6cd283d1181c00d71a5"
server_key_iv: "c6124e18947cdf7938ac327edbd2941d"
server_key_version: "encryptionkey@0"
I20260812 16:43:17.339520 19366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:43:17.343226 19382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:17.344594 19366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 16:43:17.344741 19366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "e085854173a44ca9b4b72f6c17962ed2"
format_stamp: "Formatted at 2026-08-12 16:43:17 on dist-test-slave-f7th"
server_key: "9a6ee2328d55a6cd283d1181c00d71a5"
server_key_iv: "c6124e18947cdf7938ac327edbd2941d"
server_key_version: "encryptionkey@0"
I20260812 16:43:17.344935 19366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:17.382604 19366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:17.386930 19366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:17.387207 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:17.388288 19366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:17.389662 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:17.389734 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:17.389854 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:17.389902 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:17.421489 19366 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:34791
I20260812 16:43:17.421557 19494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:34791 every 8 connection(s)
I20260812 16:43:17.422876 19366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:43:17.427448 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 19366
I20260812 16:43:17.427690 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:43:17.428248 31990 external_mini_cluster.cc:1428] Setting key b044c818a77f8ce702173babea275b8f
Aug 12 16:43:17 dist-test-slave-f7th krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552997, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.130@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:17.445879 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:17.425606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:48389 (local address 127.31.61.190:41577)
0812 16:43:17.426099 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:17.426107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:17.427168 (+ 1061us) server_negotiation.cc:407] Connection header received
0812 16:43:17.429448 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:17.429453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:17.429547 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:17.429644 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:17.431551 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.432344 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.434592 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.435036 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.435141 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:17.439281 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:17.439287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:17.439290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:17.439337 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:17.442562 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:17.443428 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:17.443433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:17.443436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:17.443659 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:17.444122 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:17.444126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:17.444129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:17.444461 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:17.444589 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:17.445301 (+ 712us) server_negotiation.cc:299] Negotiation successful
0812 16:43:17.445530 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":225,"threads_started":1}
I20260812 16:43:17.446902 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:17.426236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:41577 (local address 127.31.61.130:48389)
0812 16:43:17.426950 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:17.426994 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:17.429181 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:17.429900 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:17.429908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:17.430515 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:17.431327 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:17.431353 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.432600 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.432605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:17.434402 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:17.434417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.435278 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.435285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:17.435557 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:17.436317 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:17.436351 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:17.439007 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:17.442729 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:17.442737 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:17.442751 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:17.443272 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:17.443795 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:17.443800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:17.443802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:17.444017 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:17.444588 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:17.444595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:17.444939 (+ 344us) client_negotiation.cc:769] Sending connection context
0812 16:43:17.445181 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 16:43:17.445721 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":285,"threads_started":1}
I20260812 16:43:17.448378 19495 heartbeater.cc:347] Connected to a master server at 127.31.61.190:41577
I20260812 16:43:17.448731 19495 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:17.449664 19495 heartbeater.cc:511] Master 127.31.61.190:41577 requested a full tablet report, sending...
I20260812 16:43:17.451458 19169 ts_manager.cc:194] Registered new tserver with Master: e085854173a44ca9b4b72f6c17962ed2 (127.31.61.130:34791)
I20260812 16:43:17.452361 19169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.130@KRBTEST.COM'} at 127.31.61.130:48389
WARNING: no policy specified for kudu/127.31.61.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
Entry for principal kudu/127.31.61.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab.
I20260812 16:43:17.491117 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41577
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:17.817030 19501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:17.817341 19501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:17.817482 19501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:17.824987 19501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:17.825127 19501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:17.825284 19501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:43:17.835084 19501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.131.keytab
--principal=kudu/127.31.61.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:17.836447 19501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:17.837762 19501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:17.851329 19507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:17.851363 19510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:17.851404 19508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:17.852155 19501 server_base.cc:1034] running on GCE node
I20260812 16:43:17.852807 19501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:17.853755 19501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:17.854954 19501 hybrid_clock.cc:648] HybridClock initialized: now 1786552997854897 us; error 49 us; skew 500 ppm
I20260812 16:43:17.855244 19501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:17 dist-test-slave-f7th krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552997, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:17.858188 19501 init.cc:377] Logged in from keytab as kudu/127.31.61.131@KRBTEST.COM (short username kudu)
I20260812 16:43:17.859663 19501 webserver.cc:460] Webserver started at http://127.31.61.131:46565/ using document root <none> and password file <none>
I20260812 16:43:17.860332 19501 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:17.860455 19501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:17.860767 19501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:17.863727 19501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "6bedd557d14745f491f2a9075cf3897a"
format_stamp: "Formatted at 2026-08-12 16:43:17 on dist-test-slave-f7th"
server_key: "76a7fe505836824036374d2934122a42"
server_key_iv: "ab539198ae75ef79cc685b99b913a774"
server_key_version: "encryptionkey@0"
I20260812 16:43:17.864476 19501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "6bedd557d14745f491f2a9075cf3897a"
format_stamp: "Formatted at 2026-08-12 16:43:17 on dist-test-slave-f7th"
server_key: "76a7fe505836824036374d2934122a42"
server_key_iv: "ab539198ae75ef79cc685b99b913a774"
server_key_version: "encryptionkey@0"
I20260812 16:43:17.869524 19501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:43:17.872991 19517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:17.874312 19501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:43:17.874482 19501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "6bedd557d14745f491f2a9075cf3897a"
format_stamp: "Formatted at 2026-08-12 16:43:17 on dist-test-slave-f7th"
server_key: "76a7fe505836824036374d2934122a42"
server_key_iv: "ab539198ae75ef79cc685b99b913a774"
server_key_version: "encryptionkey@0"
I20260812 16:43:17.874675 19501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:17.891038 19501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:17.895259 19501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:17.895547 19501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:17.896541 19501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:17.897899 19501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:17.897986 19501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:17.898108 19501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:17.898167 19501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:17.920861 19360 heartbeater.cc:503] Master 127.31.61.190:41577 was elected leader, sending a full tablet report...
I20260812 16:43:17.930423 19501 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:40067
I20260812 16:43:17.930457 19629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:40067 every 8 connection(s)
I20260812 16:43:17.931731 19501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:43:17.936422 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 19501
I20260812 16:43:17.936604 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:43:17.937036 31990 external_mini_cluster.cc:1428] Setting key 5c8dd47a721ca86a1c1d67031e380068
Aug 12 16:43:17 dist-test-slave-f7th krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552997, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.131@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:17.952188 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:17.934152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:41327 (local address 127.31.61.190:41577)
0812 16:43:17.934317 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:17.934322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:17.935350 (+ 1028us) server_negotiation.cc:407] Connection header received
0812 16:43:17.937236 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:17.937241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:17.937341 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:17.937428 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:17.939210 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.940105 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.941694 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.942078 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.942190 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:17.945629 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:17.945634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:17.945637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:17.945672 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:17.948609 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:17.949357 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:17.949363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:17.949367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:17.949504 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:17.950076 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:17.950101 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:17.950104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:17.950480 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:17.950659 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:17.951654 (+ 995us) server_negotiation.cc:299] Negotiation successful
0812 16:43:17.951881 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:43:17.953158 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:17.934525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:41577 (local address 127.31.61.131:41327)
0812 16:43:17.935203 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:17.935241 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:17.936962 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:17.937700 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:17.937712 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:17.938349 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:17.939002 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:17.939022 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.940262 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.940269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:17.941553 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:17.941565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.942298 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.942305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:17.942537 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:17.943190 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:17.943214 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:17.945399 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:17.948763 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:17.948771 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:17.948786 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:17.949233 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:17.949639 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:17.949643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:17.949646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:17.949918 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:17.950631 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:17.950638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:17.951123 (+ 485us) client_negotiation.cc:769] Sending connection context
0812 16:43:17.951423 (+ 300us) client_negotiation.cc:240] Negotiation successful
0812 16:43:17.952001 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":253,"threads_started":1}
I20260812 16:43:17.954632 19630 heartbeater.cc:347] Connected to a master server at 127.31.61.190:41577
I20260812 16:43:17.954941 19630 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:17.955705 19630 heartbeater.cc:511] Master 127.31.61.190:41577 requested a full tablet report, sending...
I20260812 16:43:17.957074 19169 ts_manager.cc:194] Registered new tserver with Master: 6bedd557d14745f491f2a9075cf3897a (127.31.61.131:40067)
I20260812 16:43:17.957847 19169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.61.131@KRBTEST.COM'} at 127.31.61.131:41327
I20260812 16:43:17.964795 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:43:17.981441 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:17.968810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.31.61.190:41577)
0812 16:43:17.968972 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:17.968976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:17.969427 (+ 451us) server_negotiation.cc:407] Connection header received
0812 16:43:17.969553 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:17.969557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:17.969630 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:17.969712 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:17.971095 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.971901 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.973219 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:17.973642 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:17.973734 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:17.976301 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:17.976306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:17.976310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:17.976344 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:17.978924 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:17.979694 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:17.979699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:17.979701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:17.979841 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:17.980149 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:17.980152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:17.980154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:17.980459 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:17.980692 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:17.980989 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 16:43:17.981175 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:43:17.985721 19169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:43:17.992689 19169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:43:18.027074 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.005399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:34791 (local address 127.0.0.1:57322)
0812 16:43:18.018173 (+ 12774us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:18.018230 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:18.018467 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:18.019529 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:18.019538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:18.019575 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:18.020033 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.020047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.021751 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.021757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.024331 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.024346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.025961 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.025968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.026202 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.026273 (+ 71us) client_negotiation.cc:769] Sending connection context
0812 16:43:18.026380 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 16:43:18.026588 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12043,"mutex_wait_us":31,"spinlock_wait_cycles":2304,"thread_start_us":12093,"threads_started":1}
I20260812 16:43:18.027529 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.005577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:42181 (local address 127.0.0.1:60340)
0812 16:43:18.018178 (+ 12601us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:18.018230 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:18.018467 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:18.020251 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:18.020256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:18.020286 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:18.020722 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.020733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.022907 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.022912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.024556 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.024570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.026591 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.026597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.026807 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.026836 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 16:43:18.026932 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 16:43:18.027069 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12351,"mutex_wait_us":3,"spinlock_wait_cycles":18944,"thread_start_us":11737,"threads_started":1}
I20260812 16:43:18.027527 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.005622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.31.61.130:34791)
0812 16:43:18.018020 (+ 12398us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:18.018029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:18.018438 (+ 409us) server_negotiation.cc:407] Connection header received
0812 16:43:18.018757 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:18.018766 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:18.019048 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:18.019240 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:18.020214 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.021568 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.024540 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.025721 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.026769 (+ 1048us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.026970 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:18.027052 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 16:43:18.027227 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12201,"thread_start_us":12074,"threads_started":1}
I20260812 16:43:18.027618 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.005948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.31.61.129:42181)
0812 16:43:18.016850 (+ 10902us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:18.016862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:18.018366 (+ 1504us) server_negotiation.cc:407] Connection header received
0812 16:43:18.019523 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:18.019532 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:18.019791 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:18.019986 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:18.020964 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.022775 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.024720 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.026056 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.026206 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.026446 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:18.027120 (+ 674us) server_negotiation.cc:299] Negotiation successful
0812 16:43:18.027296 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10654,"thread_start_us":10925,"threads_started":1}
I20260812 16:43:18.031143 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.019974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:40067 (local address 127.0.0.1:42062)
0812 16:43:18.021253 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:18.021271 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:18.021532 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:18.022147 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:18.022153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:18.022180 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:18.023239 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.023252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.025147 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.025152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.028337 (+ 3185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.028352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.030143 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.030150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.030320 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.030354 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:43:18.030597 (+ 243us) client_negotiation.cc:240] Negotiation successful
0812 16:43:18.030741 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":1184,"thread_start_us":1016,"threads_started":1}
I20260812 16:43:18.031203 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.019295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.31.61.131:40067)
0812 16:43:18.019964 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:18.019972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:18.021494 (+ 1522us) server_negotiation.cc:407] Connection header received
0812 16:43:18.021713 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:18.021721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:18.021951 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:18.022152 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:18.023451 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.025004 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.028511 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.029927 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.030062 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.030309 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:18.030708 (+ 399us) server_negotiation.cc:299] Negotiation successful
0812 16:43:18.030905 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":334,"threads_started":1}
I20260812 16:43:18.031481 19430 tablet_service.cc:1467] Processing CreateTablet for tablet c4d8d3eb8516476e80c7e4b03311e897 (DEFAULT_TABLE table=test-table [id=1f4eb0266fae48dfb10a1faf5f2ce96b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:18.031510 19295 tablet_service.cc:1467] Processing CreateTablet for tablet c4d8d3eb8516476e80c7e4b03311e897 (DEFAULT_TABLE table=test-table [id=1f4eb0266fae48dfb10a1faf5f2ce96b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:18.032940 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4d8d3eb8516476e80c7e4b03311e897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:18.032961 19295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4d8d3eb8516476e80c7e4b03311e897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:18.035218 19565 tablet_service.cc:1467] Processing CreateTablet for tablet c4d8d3eb8516476e80c7e4b03311e897 (DEFAULT_TABLE table=test-table [id=1f4eb0266fae48dfb10a1faf5f2ce96b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:18.036880 19565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4d8d3eb8516476e80c7e4b03311e897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:18.045300 19649 tablet_bootstrap.cc:492] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: Bootstrap starting.
I20260812 16:43:18.050187 19649 tablet_bootstrap.cc:654] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:18.051642 19649 log.cc:824] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:18.052961 19650 tablet_bootstrap.cc:492] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: Bootstrap starting.
I20260812 16:43:18.054809 19651 tablet_bootstrap.cc:492] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: Bootstrap starting.
I20260812 16:43:18.056560 19649 tablet_bootstrap.cc:492] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: No bootstrap required, opened a new log
I20260812 16:43:18.056864 19650 tablet_bootstrap.cc:654] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:18.056867 19649 ts_tablet_manager.cc:1397] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260812 16:43:18.058214 19650 log.cc:824] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:18.058516 19651 tablet_bootstrap.cc:654] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:18.059584 19651 log.cc:824] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:18.061316 19650 tablet_bootstrap.cc:492] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: No bootstrap required, opened a new log
I20260812 16:43:18.061596 19650 ts_tablet_manager.cc:1397] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 16:43:18.062579 19651 tablet_bootstrap.cc:492] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: No bootstrap required, opened a new log
I20260812 16:43:18.062844 19651 ts_tablet_manager.cc:1397] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 16:43:18.065064 19649 raft_consensus.cc:348] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } }
I20260812 16:43:18.065457 19649 raft_consensus.cc:374] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:18.065547 19649 raft_consensus.cc:729] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47841c2b4c634114bcf53564a72619ab, State: Initialized, Role: FOLLOWER
I20260812 16:43:18.066275 19649 consensus_queue.cc:260] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } }
I20260812 16:43:18.068477 19649 ts_tablet_manager.cc:1428] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260812 16:43:18.071893 19651 raft_consensus.cc:348] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } }
I20260812 16:43:18.072311 19651 raft_consensus.cc:374] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:18.072415 19651 raft_consensus.cc:729] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bedd557d14745f491f2a9075cf3897a, State: Initialized, Role: FOLLOWER
I20260812 16:43:18.072440 19650 raft_consensus.cc:348] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } }
I20260812 16:43:18.072875 19650 raft_consensus.cc:374] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:18.073001 19650 raft_consensus.cc:729] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e085854173a44ca9b4b72f6c17962ed2, State: Initialized, Role: FOLLOWER
I20260812 16:43:18.073047 19651 consensus_queue.cc:260] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } }
I20260812 16:43:18.073539 19650 consensus_queue.cc:260] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } }
I20260812 16:43:18.075398 19495 heartbeater.cc:503] Master 127.31.61.190:41577 was elected leader, sending a full tablet report...
I20260812 16:43:18.075423 19630 heartbeater.cc:503] Master 127.31.61.190:41577 was elected leader, sending a full tablet report...
I20260812 16:43:18.075736 19651 ts_tablet_manager.cc:1428] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260812 16:43:18.075914 19650 ts_tablet_manager.cc:1428] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260812 16:43:18.100997 19657 raft_consensus.cc:482] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:43:18.101313 19657 raft_consensus.cc:504] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } }
I20260812 16:43:18.103005 19657 leader_election.cc:288] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47841c2b4c634114bcf53564a72619ab (127.31.61.129:42181), 6bedd557d14745f491f2a9075cf3897a (127.31.61.131:40067)
I20260812 16:43:18.110313 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.103615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:34287 (local address 127.31.61.129:42181)
0812 16:43:18.103839 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:18.103846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:18.104764 (+ 918us) server_negotiation.cc:407] Connection header received
0812 16:43:18.104921 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:18.104924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:18.104983 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:18.105065 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:18.105681 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.106608 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.108535 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.109322 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.109476 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.109552 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:18.109942 (+ 390us) server_negotiation.cc:299] Negotiation successful
0812 16:43:18.110074 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 16:43:18.110327 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.103757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:34373 (local address 127.31.61.131:40067)
0812 16:43:18.103976 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:18.103981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:18.104658 (+ 677us) server_negotiation.cc:407] Connection header received
0812 16:43:18.104892 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:18.104898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:18.104964 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:18.105030 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:18.105636 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.106495 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.108560 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.109323 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.109474 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.109552 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:18.109952 (+ 400us) server_negotiation.cc:299] Negotiation successful
0812 16:43:18.110085 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 16:43:18.110538 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.103266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:40067 (local address 127.31.61.130:34373)
0812 16:43:18.104549 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:18.104565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:18.104715 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:18.105071 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:18.105075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:18.105099 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:18.105482 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.105496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.106625 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.106630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.108402 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.108423 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.109608 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.109614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.109803 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.109825 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 16:43:18.109908 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 16:43:18.110033 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":56,"thread_start_us":1053,"threads_started":1}
I20260812 16:43:18.110570 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.103285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:42181 (local address 127.31.61.130:34287)
0812 16:43:18.104665 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:18.104678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:18.104791 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:18.105102 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:18.105107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:18.105130 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:18.105512 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.105522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.106756 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.106760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.108374 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:18.108391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.109565 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.109572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:18.109745 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.109804 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 16:43:18.109878 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 16:43:18.110006 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"mutex_wait_us":50,"thread_start_us":819,"threads_started":1}
I20260812 16:43:18.111521 19585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d8d3eb8516476e80c7e4b03311e897" candidate_uuid: "e085854173a44ca9b4b72f6c17962ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bedd557d14745f491f2a9075cf3897a" is_pre_election: true
I20260812 16:43:18.111546 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d8d3eb8516476e80c7e4b03311e897" candidate_uuid: "e085854173a44ca9b4b72f6c17962ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47841c2b4c634114bcf53564a72619ab" is_pre_election: true
I20260812 16:43:18.112075 19315 raft_consensus.cc:2454] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e085854173a44ca9b4b72f6c17962ed2 in term 0.
I20260812 16:43:18.112076 19585 raft_consensus.cc:2454] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e085854173a44ca9b4b72f6c17962ed2 in term 0.
I20260812 16:43:18.112665 19384 leader_election.cc:302] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bedd557d14745f491f2a9075cf3897a, e085854173a44ca9b4b72f6c17962ed2; no voters:
I20260812 16:43:18.113050 19657 raft_consensus.cc:2781] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:43:18.113225 19657 raft_consensus.cc:482] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:43:18.113328 19657 raft_consensus.cc:3037] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:18.115482 19657 raft_consensus.cc:504] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } }
I20260812 16:43:18.116158 19657 leader_election.cc:288] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [CANDIDATE]: Term 1 election: Requested vote from peers 47841c2b4c634114bcf53564a72619ab (127.31.61.129:42181), 6bedd557d14745f491f2a9075cf3897a (127.31.61.131:40067)
I20260812 16:43:18.116508 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d8d3eb8516476e80c7e4b03311e897" candidate_uuid: "e085854173a44ca9b4b72f6c17962ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47841c2b4c634114bcf53564a72619ab"
I20260812 16:43:18.116715 19315 raft_consensus.cc:3037] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:18.116834 19585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d8d3eb8516476e80c7e4b03311e897" candidate_uuid: "e085854173a44ca9b4b72f6c17962ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bedd557d14745f491f2a9075cf3897a"
I20260812 16:43:18.117020 19585 raft_consensus.cc:3037] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:18.119124 19315 raft_consensus.cc:2454] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e085854173a44ca9b4b72f6c17962ed2 in term 1.
I20260812 16:43:18.119427 19585 raft_consensus.cc:2454] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e085854173a44ca9b4b72f6c17962ed2 in term 1.
I20260812 16:43:18.119517 19383 leader_election.cc:302] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47841c2b4c634114bcf53564a72619ab, e085854173a44ca9b4b72f6c17962ed2; no voters:
I20260812 16:43:18.119880 19657 raft_consensus.cc:2781] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:18.120311 19657 raft_consensus.cc:686] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 1 LEADER]: Becoming Leader. State: Replica: e085854173a44ca9b4b72f6c17962ed2, State: Running, Role: LEADER
I20260812 16:43:18.120757 19657 consensus_queue.cc:237] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } }
I20260812 16:43:18.126324 19168 catalog_manager.cc:5495] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e085854173a44ca9b4b72f6c17962ed2 (127.31.61.130). New cstate: current_term: 1 leader_uuid: "e085854173a44ca9b4b72f6c17962ed2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e085854173a44ca9b4b72f6c17962ed2" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 34791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 } health_report { overall_health: UNKNOWN } } }
W20260812 16:43:18.140974 19361 tablet.cc:2369] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:18.163094 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.157871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.31.61.130:34791)
0812 16:43:18.158081 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:18.158086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:18.158106 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:43:18.158279 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:18.158284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:18.158367 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:18.158511 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:43:18.158879 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.159589 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.160938 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.161358 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.161483 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.161943 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:43:18.162433 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:43:18.162688 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:18.162742 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 16:43:18.162824 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 16:43:18.171267 19585 raft_consensus.cc:1261] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [term 1 FOLLOWER]: Refusing update from remote peer e085854173a44ca9b4b72f6c17962ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:18.171265 19315 raft_consensus.cc:1261] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [term 1 FOLLOWER]: Refusing update from remote peer e085854173a44ca9b4b72f6c17962ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:18.172236 19660 consensus_queue.cc:1035] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bedd557d14745f491f2a9075cf3897a" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 40067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:18.172598 19657 consensus_queue.cc:1035] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47841c2b4c634114bcf53564a72619ab" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:43:18.174971 19496 tablet.cc:2369] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:43:18.183394 19631 tablet.cc:2369] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:18.186781 19663 mvcc.cc:205] Tried to move back new op lower bound from 7317721080499597312 to 7317721080307531776. Current Snapshot: MvccSnapshot[applied={T|T < 7317721080499597312}]
I20260812 16:43:18.201912 19169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:18.202244 19169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:18.206110 19169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc113fc59d254b83a0bb493cd941428d: Sending DeleteTablet for 3 replicas of tablet c4d8d3eb8516476e80c7e4b03311e897
I20260812 16:43:18.207345 19295 tablet_service.cc:1514] Processing DeleteTablet for tablet c4d8d3eb8516476e80c7e4b03311e897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:18 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:60340
I20260812 16:43:18.207350 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet c4d8d3eb8516476e80c7e4b03311e897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:18 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:57322
I20260812 16:43:18.207571 19565 tablet_service.cc:1514] Processing DeleteTablet for tablet c4d8d3eb8516476e80c7e4b03311e897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:18 UTC) from {username='kudu', principal='kudu/127.31.61.190@KRBTEST.COM'} at 127.0.0.1:42062
I20260812 16:43:18.209883 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 19231
I20260812 16:43:18.219797 19674 tablet_replica.cc:330] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: stopping tablet replica
I20260812 16:43:18.220067 19673 tablet_replica.cc:330] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: stopping tablet replica
I20260812 16:43:18.220342 19674 raft_consensus.cc:2229] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:18.220516 19673 raft_consensus.cc:2229] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:43:18.220945 19674 raft_consensus.cc:2258] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:18.221414 19673 raft_consensus.cc:2258] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:18.223459 19672 tablet_replica.cc:330] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: stopping tablet replica
I20260812 16:43:18.223656 19674 ts_tablet_manager.cc:1905] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:18.224045 19672 raft_consensus.cc:2229] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:18.224618 19672 raft_consensus.cc:2258] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:18.226495 19673 ts_tablet_manager.cc:1905] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:18.230527 19674 ts_tablet_manager.cc:1918] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:18.230708 19674 log.cc:1196] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/c4d8d3eb8516476e80c7e4b03311e897
I20260812 16:43:18.231261 19674 ts_tablet_manager.cc:1939] T c4d8d3eb8516476e80c7e4b03311e897 P 6bedd557d14745f491f2a9075cf3897a: Deleting consensus metadata
I20260812 16:43:18.233280 19154 catalog_manager.cc:4841] TS 6bedd557d14745f491f2a9075cf3897a (127.31.61.131:40067): tablet c4d8d3eb8516476e80c7e4b03311e897 (table test-table [id=1f4eb0266fae48dfb10a1faf5f2ce96b]) successfully deleted
I20260812 16:43:18.233351 19672 ts_tablet_manager.cc:1905] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:18.234365 19673 ts_tablet_manager.cc:1918] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:18.234534 19673 log.cc:1196] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/c4d8d3eb8516476e80c7e4b03311e897
I20260812 16:43:18.235000 19673 ts_tablet_manager.cc:1939] T c4d8d3eb8516476e80c7e4b03311e897 P e085854173a44ca9b4b72f6c17962ed2: Deleting consensus metadata
I20260812 16:43:18.236708 19154 catalog_manager.cc:4841] TS e085854173a44ca9b4b72f6c17962ed2 (127.31.61.130:34791): tablet c4d8d3eb8516476e80c7e4b03311e897 (table test-table [id=1f4eb0266fae48dfb10a1faf5f2ce96b]) successfully deleted
Aug 12 16:43:18 dist-test-slave-f7th krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.129@KRBTEST.COM
I20260812 16:43:18.240469 19672 ts_tablet_manager.cc:1918] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:18.240690 19672 log.cc:1196] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786552676633950-31990-0/minicluster-data/ts-0/wal/wals/c4d8d3eb8516476e80c7e4b03311e897
I20260812 16:43:18.241274 19672 ts_tablet_manager.cc:1939] T c4d8d3eb8516476e80c7e4b03311e897 P 47841c2b4c634114bcf53564a72619ab: Deleting consensus metadata
I20260812 16:43:18.243181 19153 catalog_manager.cc:4841] TS 47841c2b4c634114bcf53564a72619ab (127.31.61.129:42181): tablet c4d8d3eb8516476e80c7e4b03311e897 (table test-table [id=1f4eb0266fae48dfb10a1faf5f2ce96b]) successfully deleted
I20260812 16:43:18.244774 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.211313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.31.61.129:42181)
0812 16:43:18.211497 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:18.211503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:18.222662 (+ 11159us) server_negotiation.cc:407] Connection header received
0812 16:43:18.222927 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:18.222937 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:18.223031 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:18.223158 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:18.225102 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.226245 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.228266 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.228908 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.229010 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.238637 (+ 9627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:18.238649 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:18.238663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:18.238703 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:18.241857 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:18.242605 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:18.242612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:18.242615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:18.242764 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:18.243157 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:18.243161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:18.243163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:18.243552 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:18.243909 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:18.244222 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 16:43:18.244438 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:43:18.245352 19355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:18.338447 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 19231
I20260812 16:43:18.360033 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 19366
Aug 12 16:43:18 dist-test-slave-f7th krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.130@KRBTEST.COM
I20260812 16:43:18.376911 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.360877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.31.61.130:34791)
0812 16:43:18.361029 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:18.361037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:18.361230 (+ 193us) server_negotiation.cc:407] Connection header received
0812 16:43:18.361416 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:18.361422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:18.361519 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:18.361647 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:18.362889 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.363849 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.365169 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.365605 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.365724 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.370436 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:18.370455 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:18.370472 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:18.370524 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:18.373771 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:18.374752 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:18.374760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:18.374763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:18.374922 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:18.375311 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:18.375314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:18.375316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:18.375795 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:18.376040 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:18.376407 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 16:43:18.376608 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 16:43:18.377586 19490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:18.492820 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 19366
I20260812 16:43:18.509500 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 19501
Aug 12 16:43:18 dist-test-slave-f7th krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786552995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.131@KRBTEST.COM
I20260812 16:43:18.525333 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:18.510218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.31.61.131:40067)
0812 16:43:18.510413 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:18.510418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:18.510437 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:43:18.510577 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:18.510582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:18.510653 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:18.510736 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:18.512039 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.512820 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.514173 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:18.514615 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:18.514775 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:18.519495 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:18.519520 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:18.519538 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:18.519634 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:18.522371 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:18.523189 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:18.523197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:18.523200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:18.523345 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:18.523771 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:18.523776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:18.523778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:18.524156 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:18.524439 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:18.524785 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 16:43:18.525013 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:43:18.525978 19625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:18.613515 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 19501
I20260812 16:43:18.628204 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 19136
I20260812 16:43:18.629330 19199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:18.740154 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 19136
2026-08-12T16:43:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4258 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T16:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:18Z Disabled control of system clock
I20260812 16:43:18.817306 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:33491
--webserver_interface=127.31.61.190
--webserver_port=0
--builtin_ntp_servers=127.31.61.148:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:33491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:43:19.156955 19683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:19.157279 19683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:19.157385 19683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:19.165259 19683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:19.165413 19683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:19.165465 19683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:19.165547 19683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:19.165602 19683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:19.175329 19683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:33491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:33491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:19.176714 19683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:19.178660 19683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:19.190680 19688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:19.190653 19689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:19.190708 19691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:19.191107 19683 server_base.cc:1034] running on GCE node
I20260812 16:43:19.192173 19683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:19.193534 19683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:19.194738 19683 hybrid_clock.cc:648] HybridClock initialized: now 1786552999194687 us; error 40 us; skew 500 ppm
I20260812 16:43:19.195080 19683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:19.197332 19683 webserver.cc:460] Webserver started at http://127.31.61.190:34877/ using document root <none> and password file <none>
I20260812 16:43:19.197989 19683 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:19.198098 19683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:19.198396 19683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:19.201386 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "4dde382b090a4f35afe28fe7b8f82cd6"
format_stamp: "Formatted at 2026-08-12 16:43:19 on dist-test-slave-f7th"
server_key: "2a10a5975392af258e2dc5e53489af4c"
server_key_iv: "626d00fdd6b65ca0834c9a578d0b9471"
server_key_version: "encryptionkey@0"
I20260812 16:43:19.202193 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "4dde382b090a4f35afe28fe7b8f82cd6"
format_stamp: "Formatted at 2026-08-12 16:43:19 on dist-test-slave-f7th"
server_key: "2a10a5975392af258e2dc5e53489af4c"
server_key_iv: "626d00fdd6b65ca0834c9a578d0b9471"
server_key_version: "encryptionkey@0"
I20260812 16:43:19.207266 19683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 16:43:19.211258 19697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:19.212857 19683 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:43:19.213119 19683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "4dde382b090a4f35afe28fe7b8f82cd6"
format_stamp: "Formatted at 2026-08-12 16:43:19 on dist-test-slave-f7th"
server_key: "2a10a5975392af258e2dc5e53489af4c"
server_key_iv: "626d00fdd6b65ca0834c9a578d0b9471"
server_key_version: "encryptionkey@0"
I20260812 16:43:19.213348 19683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:19.253208 19683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:19.254114 19683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:19.254393 19683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:19.272446 19683 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:33491
I20260812 16:43:19.272475 19748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:33491 every 8 connection(s)
I20260812 16:43:19.273877 19683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:19.274276 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 19683
I20260812 16:43:19.274442 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:19.274966 31990 external_mini_cluster.cc:1428] Setting key 003a8fbd79b8850fa407efcf1ea38566
I20260812 16:43:19.278294 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:19.288920 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:19.277008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.31.61.190:33491)
0812 16:43:19.278003 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:19.278014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:19.278058 (+ 44us) server_negotiation.cc:407] Connection header received
0812 16:43:19.279848 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:19.279871 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:19.280325 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:19.280719 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:19.281311 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:19.282761 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:19.284279 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:19.285134 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:19.285318 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:19.286496 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:19.286503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:19.286519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:19.287122 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:19.287206 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:19.287231 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:19.287341 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:19.287592 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 16:43:19.287954 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":370,"threads_started":1}
I20260812 16:43:19.289738 19749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6: Bootstrap starting.
I20260812 16:43:19.293119 19749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:19.294071 19749 log.cc:824] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:19.297031 19749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6: No bootstrap required, opened a new log
I20260812 16:43:19.304788 19749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dde382b090a4f35afe28fe7b8f82cd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33491 } }
I20260812 16:43:19.305181 19749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:19.305253 19749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dde382b090a4f35afe28fe7b8f82cd6, State: Initialized, Role: FOLLOWER
I20260812 16:43:19.305775 19749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dde382b090a4f35afe28fe7b8f82cd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33491 } }
I20260812 16:43:19.306003 19749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:19.306108 19749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:19.306231 19749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:19.308704 19749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dde382b090a4f35afe28fe7b8f82cd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33491 } }
I20260812 16:43:19.309187 19749 leader_election.cc:302] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dde382b090a4f35afe28fe7b8f82cd6; no voters:
I20260812 16:43:19.309728 19749 leader_election.cc:288] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:19.309885 19754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:19.310535 19754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [term 1 LEADER]: Becoming Leader. State: Replica: 4dde382b090a4f35afe28fe7b8f82cd6, State: Running, Role: LEADER
I20260812 16:43:19.311055 19754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dde382b090a4f35afe28fe7b8f82cd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33491 } }
I20260812 16:43:19.311772 19749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:19.314013 19756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dde382b090a4f35afe28fe7b8f82cd6. Latest consensus state: current_term: 1 leader_uuid: "4dde382b090a4f35afe28fe7b8f82cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dde382b090a4f35afe28fe7b8f82cd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33491 } } }
I20260812 16:43:19.314414 19756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:19.314488 19755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dde382b090a4f35afe28fe7b8f82cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dde382b090a4f35afe28fe7b8f82cd6" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 33491 } } }
I20260812 16:43:19.314822 19755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:19.318226 19763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:19.323734 19763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:19.332858 19763 catalog_manager.cc:1347] Generated new cluster ID: a58e0f77af57467ea1835d7a5efe30d5
I20260812 16:43:19.332973 19763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:19.369949 19763 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:19.371361 19763 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:19.380195 19763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6: Generated new TSK 0
I20260812 16:43:19.381031 19763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:19.390261 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.129:0
--local_ip_for_outbound_sockets=127.31.61.129
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33491
--builtin_ntp_servers=127.31.61.148:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 16:43:19.748430 19774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:19.748733 19774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:19.748839 19774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:19.756189 19774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:19.756299 19774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:19.756438 19774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.129
I20260812 16:43:19.765813 19774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.61.129
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.129
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:19.767113 19774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:19.768870 19774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:19.782260 19782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:19.782284 19780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:19.782320 19779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:19.782855 19774 server_base.cc:1034] running on GCE node
I20260812 16:43:19.783555 19774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:19.784577 19774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:19.785910 19774 hybrid_clock.cc:648] HybridClock initialized: now 1786552999785851 us; error 40 us; skew 500 ppm
I20260812 16:43:19.786340 19774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:19.788913 19774 webserver.cc:460] Webserver started at http://127.31.61.129:41747/ using document root <none> and password file <none>
I20260812 16:43:19.789614 19774 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:19.789700 19774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:19.789965 19774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:19.793000 19774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/data/instance:
uuid: "607c1ddf20d34363bbaf48dc42e7cc3e"
format_stamp: "Formatted at 2026-08-12 16:43:19 on dist-test-slave-f7th"
server_key: "b76c9a9aabcef10abc0c035439717605"
server_key_iv: "55ceafa85fe13d54348eda6af2f18d52"
server_key_version: "encryptionkey@0"
I20260812 16:43:19.793769 19774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance:
uuid: "607c1ddf20d34363bbaf48dc42e7cc3e"
format_stamp: "Formatted at 2026-08-12 16:43:19 on dist-test-slave-f7th"
server_key: "b76c9a9aabcef10abc0c035439717605"
server_key_iv: "55ceafa85fe13d54348eda6af2f18d52"
server_key_version: "encryptionkey@0"
I20260812 16:43:19.798825 19774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:43:19.802386 19788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:19.803891 19774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:19.804055 19774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/wal
uuid: "607c1ddf20d34363bbaf48dc42e7cc3e"
format_stamp: "Formatted at 2026-08-12 16:43:19 on dist-test-slave-f7th"
server_key: "b76c9a9aabcef10abc0c035439717605"
server_key_iv: "55ceafa85fe13d54348eda6af2f18d52"
server_key_version: "encryptionkey@0"
I20260812 16:43:19.804270 19774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:19.819947 19774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:19.820822 19774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:19.821146 19774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:19.822131 19774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:19.823649 19774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:19.823757 19774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:19.823870 19774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:19.823940 19774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:19.852661 19774 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.129:40951
I20260812 16:43:19.852712 19901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.129:40951 every 8 connection(s)
I20260812 16:43:19.853931 19774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/data/info.pb
I20260812 16:43:19.861851 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 19774
I20260812 16:43:19.862020 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-0/wal/instance
I20260812 16:43:19.862601 31990 external_mini_cluster.cc:1428] Setting key 9d46b0b081e4db209626297e135b5c2f
I20260812 16:43:19.865334 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.130:0
--local_ip_for_outbound_sockets=127.31.61.130
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33491
--builtin_ntp_servers=127.31.61.148:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:43:19.866472 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:19.856626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.129:36169 (local address 127.31.61.190:33491)
0812 16:43:19.857090 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:19.857095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:19.858185 (+ 1090us) server_negotiation.cc:407] Connection header received
0812 16:43:19.859967 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:19.859971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:19.860041 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:19.860135 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:19.861168 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:19.861952 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:19.863794 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:19.864215 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:19.864324 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:19.865661 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:19.865668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:19.865672 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:19.865692 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:19.865742 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:19.865756 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:19.865846 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:19.866026 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 16:43:19.866137 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":236,"threads_started":1}
I20260812 16:43:19.867398 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:19.857171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:33491 (local address 127.31.61.129:36169)
0812 16:43:19.858033 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:19.858075 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:19.859728 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:19.860318 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:19.860327 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:19.860410 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:19.860992 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:19.861009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:19.862157 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:19.862161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:19.863665 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:19.863680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:19.864446 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:19.864453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:19.864684 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:19.865332 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:43:19.865334 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:19.865374 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:43:19.865457 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:19.865844 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:19.865852 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:19.865858 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 16:43:19.865999 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 16:43:19.866320 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":264,"threads_started":1}
I20260812 16:43:19.868966 19902 heartbeater.cc:347] Connected to a master server at 127.31.61.190:33491
I20260812 16:43:19.869258 19902 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:19.870008 19902 heartbeater.cc:511] Master 127.31.61.190:33491 requested a full tablet report, sending...
I20260812 16:43:19.872217 19714 ts_manager.cc:194] Registered new tserver with Master: 607c1ddf20d34363bbaf48dc42e7cc3e (127.31.61.129:40951)
I20260812 16:43:19.874073 19714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.129:36169
W20260812 16:43:20.201591 19907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:20.201905 19907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:20.202031 19907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:20.209619 19907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:20.209749 19907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:20.209896 19907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.130
I20260812 16:43:20.219669 19907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.61.130
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.130
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:20.221017 19907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:20.222920 19907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:20.237323 19913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:20.237290 19912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:20.237255 19915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:20.237833 19907 server_base.cc:1034] running on GCE node
I20260812 16:43:20.238821 19907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:20.239964 19907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:20.241204 19907 hybrid_clock.cc:648] HybridClock initialized: now 1786553000241148 us; error 44 us; skew 500 ppm
I20260812 16:43:20.241585 19907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:20.244215 19907 webserver.cc:460] Webserver started at http://127.31.61.130:38589/ using document root <none> and password file <none>
I20260812 16:43:20.244953 19907 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:20.245065 19907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:20.245394 19907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:20.248453 19907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/data/instance:
uuid: "07028c47730e4a72ab404fa9b055c33f"
format_stamp: "Formatted at 2026-08-12 16:43:20 on dist-test-slave-f7th"
server_key: "2bb2a8afde3f69caeaa8b46d9b8a57fb"
server_key_iv: "d616f0048b290075d7650f35474c687b"
server_key_version: "encryptionkey@0"
I20260812 16:43:20.249274 19907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance:
uuid: "07028c47730e4a72ab404fa9b055c33f"
format_stamp: "Formatted at 2026-08-12 16:43:20 on dist-test-slave-f7th"
server_key: "2bb2a8afde3f69caeaa8b46d9b8a57fb"
server_key_iv: "d616f0048b290075d7650f35474c687b"
server_key_version: "encryptionkey@0"
I20260812 16:43:20.254761 19907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:43:20.258885 19921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:20.260608 19907 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 16:43:20.260802 19907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/wal
uuid: "07028c47730e4a72ab404fa9b055c33f"
format_stamp: "Formatted at 2026-08-12 16:43:20 on dist-test-slave-f7th"
server_key: "2bb2a8afde3f69caeaa8b46d9b8a57fb"
server_key_iv: "d616f0048b290075d7650f35474c687b"
server_key_version: "encryptionkey@0"
I20260812 16:43:20.261029 19907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:20.281373 19907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:20.282437 19907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:20.282758 19907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:20.284065 19907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:20.286152 19907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:20.286269 19907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:20.286373 19907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:20.286441 19907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:20.322139 19907 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.130:37403
I20260812 16:43:20.322187 20034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.130:37403 every 8 connection(s)
I20260812 16:43:20.323462 19907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/data/info.pb
I20260812 16:43:20.324218 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 19907
I20260812 16:43:20.324391 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/wal/instance
I20260812 16:43:20.324929 31990 external_mini_cluster.cc:1428] Setting key 01988285f41543e0c0829e47b1a07dd1
I20260812 16:43:20.328254 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.61.131:0
--local_ip_for_outbound_sockets=127.31.61.131
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33491
--builtin_ntp_servers=127.31.61.148:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 16:43:20.336418 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.326316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:51013 (local address 127.31.61.190:33491)
0812 16:43:20.326474 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:20.326478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:20.328065 (+ 1587us) server_negotiation.cc:407] Connection header received
0812 16:43:20.329880 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:20.329885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:20.329968 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:20.330055 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:20.331167 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.331947 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.333523 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.333953 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.334068 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.335589 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:20.335623 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:20.335628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:20.335653 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:20.335690 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:20.335705 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:20.335784 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:20.336074 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 16:43:20.336164 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 16:43:20.337567 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.326951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:33491 (local address 127.31.61.130:51013)
0812 16:43:20.327927 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:20.327964 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:20.329602 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:20.330316 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:20.330326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:20.330420 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:20.330991 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.331009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.332111 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.332115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.333365 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.333379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.334215 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.334223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.334444 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.335228 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:43:20.335230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:20.335274 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:43:20.335358 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:20.335816 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:20.335827 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:20.335836 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 16:43:20.336023 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 16:43:20.336463 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
I20260812 16:43:20.338958 20035 heartbeater.cc:347] Connected to a master server at 127.31.61.190:33491
I20260812 16:43:20.339275 20035 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:20.340227 20035 heartbeater.cc:511] Master 127.31.61.190:33491 requested a full tablet report, sending...
I20260812 16:43:20.341825 19714 ts_manager.cc:194] Registered new tserver with Master: 07028c47730e4a72ab404fa9b055c33f (127.31.61.130:37403)
I20260812 16:43:20.342631 19714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.130:51013
W20260812 16:43:20.661754 20038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:20.662061 20038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:20.662158 20038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:20.669723 20038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:20.669842 20038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:20.669984 20038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.61.131
I20260812 16:43:20.679370 20038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.61.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.61.131
--webserver_port=0
--tserver_master_addrs=127.31.61.190:33491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.61.131
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:20.680783 20038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:20.682616 20038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:20.697127 20047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:20.697196 20044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:20.697216 20045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:20.698590 20038 server_base.cc:1034] running on GCE node
I20260812 16:43:20.699268 20038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:20.700940 20038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:20.702214 20038 hybrid_clock.cc:648] HybridClock initialized: now 1786553000702142 us; error 56 us; skew 500 ppm
I20260812 16:43:20.702661 20038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 16:43:20.705827 20038 webserver.cc:460] Webserver started at http://127.31.61.131:33057/ using document root <none> and password file <none>
I20260812 16:43:20.706733 20038 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:20.706905 20038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:20.707299 20038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:20.710712 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/data/instance:
uuid: "a24cce456fda460cae506983eb11f214"
format_stamp: "Formatted at 2026-08-12 16:43:20 on dist-test-slave-f7th"
server_key: "8d7b9a0f7a0c930d6c642ebcf910e228"
server_key_iv: "0d433e743216f6bd88f5ba340dd89d65"
server_key_version: "encryptionkey@0"
I20260812 16:43:20.711591 20038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance:
uuid: "a24cce456fda460cae506983eb11f214"
format_stamp: "Formatted at 2026-08-12 16:43:20 on dist-test-slave-f7th"
server_key: "8d7b9a0f7a0c930d6c642ebcf910e228"
server_key_iv: "0d433e743216f6bd88f5ba340dd89d65"
server_key_version: "encryptionkey@0"
I20260812 16:43:20.718225 20038 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260812 16:43:20.722026 20053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:20.723548 20038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:43:20.723757 20038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/wal
uuid: "a24cce456fda460cae506983eb11f214"
format_stamp: "Formatted at 2026-08-12 16:43:20 on dist-test-slave-f7th"
server_key: "8d7b9a0f7a0c930d6c642ebcf910e228"
server_key_iv: "0d433e743216f6bd88f5ba340dd89d65"
server_key_version: "encryptionkey@0"
I20260812 16:43:20.723978 20038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:20.746896 20038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:20.747843 20038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:20.748147 20038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:20.749164 20038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 16:43:20.750742 20038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 16:43:20.750840 20038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:20.750952 20038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 16:43:20.751024 20038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:20.783406 20038 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.131:38051
I20260812 16:43:20.783492 20167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.131:38051 every 8 connection(s)
I20260812 16:43:20.784735 20038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/data/info.pb
I20260812 16:43:20.787210 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 20038
I20260812 16:43:20.787381 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/wal/instance
I20260812 16:43:20.787914 31990 external_mini_cluster.cc:1428] Setting key a751b0255026b927464e0496d33ac802
I20260812 16:43:20.797875 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.787441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.131:58241 (local address 127.31.61.190:33491)
0812 16:43:20.787718 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:20.787725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:20.789465 (+ 1740us) server_negotiation.cc:407] Connection header received
0812 16:43:20.791557 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:20.791561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:20.791660 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:20.791745 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:20.792889 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.793815 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.795336 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.795814 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.795909 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.797149 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:20.797154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:20.797157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:20.797176 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:20.797208 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:20.797219 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:20.797287 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:20.797558 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 16:43:20.797638 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:43:20.799017 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.787988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:33491 (local address 127.31.61.131:58241)
0812 16:43:20.789296 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:20.789346 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:20.791300 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:20.791985 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:20.791993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:20.792089 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:20.792698 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.792715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.793977 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.793981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.795210 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.795223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.795993 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.795998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.796212 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.796841 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 16:43:20.796844 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:20.796885 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 16:43:20.796971 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:20.797359 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:20.797369 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:20.797376 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 16:43:20.797552 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 16:43:20.797869 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":568,"threads_started":1}
I20260812 16:43:20.800532 20168 heartbeater.cc:347] Connected to a master server at 127.31.61.190:33491
I20260812 16:43:20.800865 20168 heartbeater.cc:464] Registering TS with master...
I20260812 16:43:20.801812 20168 heartbeater.cc:511] Master 127.31.61.190:33491 requested a full tablet report, sending...
I20260812 16:43:20.803494 19713 ts_manager.cc:194] Registered new tserver with Master: a24cce456fda460cae506983eb11f214 (127.31.61.131:38051)
I20260812 16:43:20.804394 19713 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.61.131:58241
I20260812 16:43:20.816210 31990 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 16:43:20.826547 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.820603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.31.61.190:33491)
0812 16:43:20.820812 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:20.820817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:20.821218 (+ 401us) server_negotiation.cc:407] Connection header received
0812 16:43:20.821356 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:20.821359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:20.821439 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:20.821539 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:20.821978 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.822887 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.824328 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.824765 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.824868 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.825899 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:20.825904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:20.825908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:20.825927 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:20.825969 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:20.825982 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:20.826053 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:20.826167 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 16:43:20.826249 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:43:20.830669 19713 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 16:43:20.837612 19713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 16:43:20.860796 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.849816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:40951 (local address 127.0.0.1:55150)
0812 16:43:20.850609 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:20.850654 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:20.850817 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:20.851872 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:20.851881 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:20.851922 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:20.852309 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.852323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.854519 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.854526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.857187 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.857209 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.859739 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.859747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.860001 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.860067 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 16:43:20.860209 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 16:43:20.860419 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":365,"threads_started":1}
I20260812 16:43:20.862207 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.849967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.31.61.129:40951)
0812 16:43:20.850534 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:20.850543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:20.850751 (+ 208us) server_negotiation.cc:407] Connection header received
0812 16:43:20.851090 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:20.851098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:20.851350 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:20.851544 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:20.852523 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.854337 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.857456 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.859083 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.859225 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.859562 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:20.861534 (+ 1972us) server_negotiation.cc:299] Negotiation successful
0812 16:43:20.861746 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":1110,"threads_started":1}
I20260812 16:43:20.871656 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.850419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.130:37403 (local address 127.0.0.1:46230)
0812 16:43:20.861217 (+ 10798us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:20.861240 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:20.861358 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:20.863697 (+ 2339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:20.863703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:20.863735 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:20.863933 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.863945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.867005 (+ 3060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.867011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.868856 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.868876 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.870685 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.870691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.870904 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.870998 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 16:43:20.871102 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 16:43:20.871251 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10642,"mutex_wait_us":42,"thread_start_us":17469,"threads_started":1}
I20260812 16:43:20.871727 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.850906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.31.61.130:37403)
0812 16:43:20.862931 (+ 12025us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:20.862942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:20.862978 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:43:20.863106 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:20.863115 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:20.863394 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:20.864136 (+ 742us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:20.864367 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.866824 (+ 2457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.869073 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.870456 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.870698 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.870941 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:20.871203 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 16:43:20.871377 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11846,"thread_start_us":11713,"threads_started":1}
I20260812 16:43:20.871079 19837 tablet_service.cc:1467] Processing CreateTablet for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8 (DEFAULT_TABLE table=test-table [id=463908c3093b45dd94f4532281d40169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:20.872841 19837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:20.875375 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.850976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:38051 (local address 127.0.0.1:47278)
0812 16:43:20.865597 (+ 14621us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:20.865623 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:20.865744 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:20.868357 (+ 2613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:20.868362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:20.868389 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:20.868827 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.868842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.870484 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.870490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.872783 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.872809 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.874620 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.874627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.874801 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.874835 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 16:43:20.874926 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 16:43:20.875069 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14477,"thread_start_us":14307,"threads_started":1}
I20260812 16:43:20.875715 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.851262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.31.61.131:38051)
0812 16:43:20.867750 (+ 16488us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:20.867759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:20.867795 (+ 36us) server_negotiation.cc:407] Connection header received
0812 16:43:20.867895 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:20.867902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:20.868160 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:20.868341 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:20.869045 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.870347 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.873392 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.874415 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.874541 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.874817 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:20.874983 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 16:43:20.875176 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16283,"thread_start_us":16139,"threads_started":1}
I20260812 16:43:20.876062 19970 tablet_service.cc:1467] Processing CreateTablet for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8 (DEFAULT_TABLE table=test-table [id=463908c3093b45dd94f4532281d40169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:20.877573 19970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:20.877568 19902 heartbeater.cc:503] Master 127.31.61.190:33491 was elected leader, sending a full tablet report...
I20260812 16:43:20.879472 20102 tablet_service.cc:1467] Processing CreateTablet for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8 (DEFAULT_TABLE table=test-table [id=463908c3093b45dd94f4532281d40169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 16:43:20.880939 20102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:20.889693 20187 tablet_bootstrap.cc:492] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e: Bootstrap starting.
I20260812 16:43:20.893090 20187 tablet_bootstrap.cc:654] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:20.894114 20187 log.cc:824] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:20.895087 20188 tablet_bootstrap.cc:492] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: Bootstrap starting.
I20260812 16:43:20.896697 20190 tablet_bootstrap.cc:492] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: Bootstrap starting.
I20260812 16:43:20.899492 20188 tablet_bootstrap.cc:654] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:20.899583 20190 tablet_bootstrap.cc:654] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:20.900590 20190 log.cc:824] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:20.900976 20188 log.cc:824] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:20.903800 20190 tablet_bootstrap.cc:492] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: No bootstrap required, opened a new log
I20260812 16:43:20.903849 20187 tablet_bootstrap.cc:492] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e: No bootstrap required, opened a new log
I20260812 16:43:20.904032 20188 tablet_bootstrap.cc:492] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: No bootstrap required, opened a new log
I20260812 16:43:20.904081 20190 ts_tablet_manager.cc:1397] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 16:43:20.904166 20187 ts_tablet_manager.cc:1397] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260812 16:43:20.904302 20188 ts_tablet_manager.cc:1397] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260812 16:43:20.912992 20190 raft_consensus.cc:348] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } }
I20260812 16:43:20.913463 20190 raft_consensus.cc:374] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:20.913591 20190 raft_consensus.cc:729] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24cce456fda460cae506983eb11f214, State: Initialized, Role: FOLLOWER
I20260812 16:43:20.914217 20190 consensus_queue.cc:260] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } }
I20260812 16:43:20.915337 20168 heartbeater.cc:503] Master 127.31.61.190:33491 was elected leader, sending a full tablet report...
I20260812 16:43:20.915371 20188 raft_consensus.cc:348] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } }
I20260812 16:43:20.915912 20188 raft_consensus.cc:374] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:20.916028 20188 raft_consensus.cc:729] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07028c47730e4a72ab404fa9b055c33f, State: Initialized, Role: FOLLOWER
I20260812 16:43:20.916072 20190 ts_tablet_manager.cc:1428] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260812 16:43:20.915978 20187 raft_consensus.cc:348] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } }
I20260812 16:43:20.916443 20187 raft_consensus.cc:374] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:20.916605 20187 raft_consensus.cc:729] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607c1ddf20d34363bbaf48dc42e7cc3e, State: Initialized, Role: FOLLOWER
I20260812 16:43:20.916854 20188 consensus_queue.cc:260] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } }
I20260812 16:43:20.917557 20187 consensus_queue.cc:260] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } }
I20260812 16:43:20.918686 20035 heartbeater.cc:503] Master 127.31.61.190:33491 was elected leader, sending a full tablet report...
I20260812 16:43:20.919518 20188 ts_tablet_manager.cc:1428] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260812 16:43:20.921156 20187 ts_tablet_manager.cc:1428] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
I20260812 16:43:20.924899 20194 raft_consensus.cc:482] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 16:43:20.925192 20194 raft_consensus.cc:504] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } }
I20260812 16:43:20.926728 20194 leader_election.cc:288] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607c1ddf20d34363bbaf48dc42e7cc3e (127.31.61.129:40951), a24cce456fda460cae506983eb11f214 (127.31.61.131:38051)
I20260812 16:43:20.934114 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.928268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:43199 (local address 127.31.61.131:38051)
0812 16:43:20.928458 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:20.928464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:20.928543 (+ 79us) server_negotiation.cc:407] Connection header received
0812 16:43:20.928768 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:20.928773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:20.928832 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:20.928897 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:20.929530 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.930546 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.932403 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.933219 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.933373 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.933447 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:20.933730 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 16:43:20.933855 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 16:43:20.934212 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.927221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.61.130:37813 (local address 127.31.61.129:40951)
0812 16:43:20.927448 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:20.927455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:20.928383 (+ 928us) server_negotiation.cc:407] Connection header received
0812 16:43:20.928475 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:20.928478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:20.928539 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:20.928609 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 16:43:20.929253 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.930343 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.932428 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.933321 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.933488 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.933568 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:20.933850 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 16:43:20.933977 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 16:43:20.934247 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.927928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.131:38051 (local address 127.31.61.130:43199)
0812 16:43:20.928458 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:20.928475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:20.928595 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:20.928941 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:20.928949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:20.928973 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:20.929360 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.929368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.930679 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.930683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.932204 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.932218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.933405 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.933412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.933578 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.933606 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 16:43:20.933690 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 16:43:20.933826 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":41,"thread_start_us":254,"threads_started":1}
I20260812 16:43:20.934309 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.927097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:40951 (local address 127.31.61.130:37813)
0812 16:43:20.927850 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:20.927872 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:20.928023 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:20.928631 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:20.928635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:20.928659 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 16:43:20.929084 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.929097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.930483 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.930488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.932218 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:20.932232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.933515 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.933522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:20.933709 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.933732 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 16:43:20.933810 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 16:43:20.933934 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":26,"thread_start_us":389,"threads_started":1}
I20260812 16:43:20.935333 19857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57157f69f41f47ef9ecddf1a5cdc6eb8" candidate_uuid: "07028c47730e4a72ab404fa9b055c33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" is_pre_election: true
I20260812 16:43:20.935328 20122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57157f69f41f47ef9ecddf1a5cdc6eb8" candidate_uuid: "07028c47730e4a72ab404fa9b055c33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24cce456fda460cae506983eb11f214" is_pre_election: true
I20260812 16:43:20.935958 20122 raft_consensus.cc:2454] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07028c47730e4a72ab404fa9b055c33f in term 0.
I20260812 16:43:20.935958 19857 raft_consensus.cc:2454] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07028c47730e4a72ab404fa9b055c33f in term 0.
I20260812 16:43:20.936673 19924 leader_election.cc:302] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07028c47730e4a72ab404fa9b055c33f, 607c1ddf20d34363bbaf48dc42e7cc3e; no voters:
I20260812 16:43:20.937023 20194 raft_consensus.cc:2781] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 16:43:20.937260 20194 raft_consensus.cc:482] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 16:43:20.937390 20194 raft_consensus.cc:3037] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:20.939553 20194 raft_consensus.cc:504] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } }
I20260812 16:43:20.940176 20194 leader_election.cc:288] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [CANDIDATE]: Term 1 election: Requested vote from peers 607c1ddf20d34363bbaf48dc42e7cc3e (127.31.61.129:40951), a24cce456fda460cae506983eb11f214 (127.31.61.131:38051)
I20260812 16:43:20.940685 19857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57157f69f41f47ef9ecddf1a5cdc6eb8" candidate_uuid: "07028c47730e4a72ab404fa9b055c33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e"
I20260812 16:43:20.940797 20122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57157f69f41f47ef9ecddf1a5cdc6eb8" candidate_uuid: "07028c47730e4a72ab404fa9b055c33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24cce456fda460cae506983eb11f214"
I20260812 16:43:20.940886 19857 raft_consensus.cc:3037] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:20.941016 20122 raft_consensus.cc:3037] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:20.943185 19857 raft_consensus.cc:2454] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07028c47730e4a72ab404fa9b055c33f in term 1.
I20260812 16:43:20.943365 20122 raft_consensus.cc:2454] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07028c47730e4a72ab404fa9b055c33f in term 1.
I20260812 16:43:20.943632 19924 leader_election.cc:302] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07028c47730e4a72ab404fa9b055c33f, 607c1ddf20d34363bbaf48dc42e7cc3e; no voters:
I20260812 16:43:20.943918 20194 raft_consensus.cc:2781] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:20.944335 20194 raft_consensus.cc:686] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 1 LEADER]: Becoming Leader. State: Replica: 07028c47730e4a72ab404fa9b055c33f, State: Running, Role: LEADER
I20260812 16:43:20.944798 20194 consensus_queue.cc:237] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } }
I20260812 16:43:20.949316 19714 catalog_manager.cc:5495] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f reported cstate change: term changed from 0 to 1, leader changed from <none> to 07028c47730e4a72ab404fa9b055c33f (127.31.61.130). New cstate: current_term: 1 leader_uuid: "07028c47730e4a72ab404fa9b055c33f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07028c47730e4a72ab404fa9b055c33f" member_type: VOTER last_known_addr { host: "127.31.61.130" port: 37403 } health_report { overall_health: HEALTHY } } }
I20260812 16:43:20.978595 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:20.973244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.31.61.130:37403)
0812 16:43:20.973541 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:20.973547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:20.973569 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:43:20.973733 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:20.973737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:20.973839 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:20.973988 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:43:20.974403 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.975157 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.976569 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:20.976993 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:20.977093 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:20.977529 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:43:20.978056 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:43:20.978211 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:20.978260 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 16:43:20.978345 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260812 16:43:20.986877 19857 raft_consensus.cc:1261] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [term 1 FOLLOWER]: Refusing update from remote peer 07028c47730e4a72ab404fa9b055c33f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:20.986878 20122 raft_consensus.cc:1261] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [term 1 FOLLOWER]: Refusing update from remote peer 07028c47730e4a72ab404fa9b055c33f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:20.987910 20199 consensus_queue.cc:1035] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [LEADER]: Connected to new peer: Peer: permanent_uuid: "607c1ddf20d34363bbaf48dc42e7cc3e" member_type: VOTER last_known_addr { host: "127.31.61.129" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:20.988253 20194 consensus_queue.cc:1035] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24cce456fda460cae506983eb11f214" member_type: VOTER last_known_addr { host: "127.31.61.131" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:20.999310 20202 mvcc.cc:205] Tried to move back new op lower bound from 7317721092031873024 to 7317721091873792000. Current Snapshot: MvccSnapshot[applied={T|T < 7317721092031873024}]
I20260812 16:43:21.002033 20204 mvcc.cc:205] Tried to move back new op lower bound from 7317721092031873024 to 7317721091873792000. Current Snapshot: MvccSnapshot[applied={T|T < 7317721092031873024}]
I20260812 16:43:21.016057 19714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:21.016343 19714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 16:43:21.020217 19714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dde382b090a4f35afe28fe7b8f82cd6: Sending DeleteTablet for 3 replicas of tablet 57157f69f41f47ef9ecddf1a5cdc6eb8
I20260812 16:43:21.021484 19970 tablet_service.cc:1514] Processing DeleteTablet for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:21 UTC) from {username='slave'} at 127.0.0.1:46230
I20260812 16:43:21.021469 20102 tablet_service.cc:1514] Processing DeleteTablet for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:21 UTC) from {username='slave'} at 127.0.0.1:47278
I20260812 16:43:21.022249 19837 tablet_service.cc:1514] Processing DeleteTablet for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 16:43:21 UTC) from {username='slave'} at 127.0.0.1:55150
I20260812 16:43:21.022876 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 19774
I20260812 16:43:21.036266 20214 tablet_replica.cc:330] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e: stopping tablet replica
I20260812 16:43:21.039403 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.026286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.31.61.129:40951)
0812 16:43:21.026467 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:21.026473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:21.026495 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:43:21.026717 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:21.026724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:21.026853 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:21.026930 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:21.027730 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:21.028992 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:21.031307 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:21.032090 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:21.032197 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:21.037259 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:21.037273 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:21.037293 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:21.038127 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:21.038198 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:21.038227 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:21.038320 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:21.038746 (+ 426us) server_negotiation.cc:299] Negotiation successful
0812 16:43:21.038917 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:43:21.040513 20214 raft_consensus.cc:2229] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:21.040997 19897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:21.041787 20213 tablet_replica.cc:330] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: stopping tablet replica
I20260812 16:43:21.042358 20213 raft_consensus.cc:2229] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 1 LEADER]: Raft consensus shutting down.
I20260812 16:43:21.043143 20213 raft_consensus.cc:2258] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:21.044332 20212 tablet_replica.cc:330] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: stopping tablet replica
I20260812 16:43:21.044812 20212 raft_consensus.cc:2229] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 16:43:21.045455 20212 raft_consensus.cc:2258] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:21.048192 20212 ts_tablet_manager.cc:1905] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:21.046658 20213 ts_tablet_manager.cc:1905] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 16:43:21.055401 20212 ts_tablet_manager.cc:1918] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:21.055577 20212 log.cc:1196] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-2/wal/wals/57157f69f41f47ef9ecddf1a5cdc6eb8
I20260812 16:43:21.056687 20212 ts_tablet_manager.cc:1939] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P a24cce456fda460cae506983eb11f214: Deleting consensus metadata
I20260812 16:43:21.059198 20213 ts_tablet_manager.cc:1918] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 16:43:21.059258 19700 catalog_manager.cc:4841] TS a24cce456fda460cae506983eb11f214 (127.31.61.131:38051): tablet 57157f69f41f47ef9ecddf1a5cdc6eb8 (table test-table [id=463908c3093b45dd94f4532281d40169]) successfully deleted
I20260812 16:43:21.059513 20213 log.cc:1196] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: Deleting WAL directory at /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786552676633950-31990-0/minicluster-data/ts-1/wal/wals/57157f69f41f47ef9ecddf1a5cdc6eb8
I20260812 16:43:21.060680 20213 ts_tablet_manager.cc:1939] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 07028c47730e4a72ab404fa9b055c33f: Deleting consensus metadata
I20260812 16:43:21.062841 19699 catalog_manager.cc:4841] TS 07028c47730e4a72ab404fa9b055c33f (127.31.61.130:37403): tablet 57157f69f41f47ef9ecddf1a5cdc6eb8 (table test-table [id=463908c3093b45dd94f4532281d40169]) successfully deleted
I20260812 16:43:21.163532 20214 raft_consensus.cc:2258] T 57157f69f41f47ef9ecddf1a5cdc6eb8 P 607c1ddf20d34363bbaf48dc42e7cc3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 16:43:21.164260 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 19774
I20260812 16:43:21.188928 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 19907
W20260812 16:43:21.188799 19700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.61.129:40951 (error 108)
I20260812 16:43:21.192948 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.191260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:40951 (local address 127.0.0.1:55176)
0812 16:43:21.192580 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:21.192751 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:40951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 16:43:21.193285 19700 catalog_manager.cc:4584] TS 607c1ddf20d34363bbaf48dc42e7cc3e (127.31.61.129:40951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57157f69f41f47ef9ecddf1a5cdc6eb8: Network error: Client connection negotiation failed: client connection to 127.31.61.129:40951: connect: Connection refused (error 111)
I20260812 16:43:21.197319 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.189918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.31.61.130:37403)
0812 16:43:21.190103 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:21.190109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:21.190129 (+ 20us) server_negotiation.cc:407] Connection header received
0812 16:43:21.190327 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:21.190337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:21.190450 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:21.190583 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:21.191242 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:21.192531 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:21.194195 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:21.194679 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:21.194804 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:21.195921 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:21.195931 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:21.195946 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:21.196549 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:21.196599 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:21.196621 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:21.196726 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:21.196869 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 16:43:21.196985 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 16:43:21.198137 20030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:21.247407 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.246845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:40951 (local address 127.0.0.1:55192)
0812 16:43:21.247115 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:21.247215 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:40951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260812 16:43:21.248476 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.248144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:40951 (local address 127.0.0.1:55206)
0812 16:43:21.248264 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:21.248337 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:40951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 16:43:21.309001 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.308480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:40951 (local address 127.0.0.1:55218)
0812 16:43:21.308718 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:21.308815 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:40951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 16:43:21.310135 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.309766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:40951 (local address 127.0.0.1:55220)
0812 16:43:21.309914 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:21.310002 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:40951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 16:43:21.312505 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 19907
I20260812 16:43:21.328859 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20038
I20260812 16:43:21.338832 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.329581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.31.61.131:38051)
0812 16:43:21.329778 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:21.329784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:21.329863 (+ 79us) server_negotiation.cc:407] Connection header received
0812 16:43:21.330071 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:21.330077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:21.330168 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:21.330268 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:21.330659 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:21.331382 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:21.332725 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:21.333130 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:21.333243 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:21.337451 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:21.337462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:21.337479 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:21.338049 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:21.338103 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:21.338126 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:21.338243 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:21.338405 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 16:43:21.338532 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 16:43:21.339536 20163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:21.390885 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.390338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:40951 (local address 127.0.0.1:55224)
0812 16:43:21.390560 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:21.390665 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:40951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 16:43:21.392078 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:21.391638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.129:40951 (local address 127.0.0.1:55230)
0812 16:43:21.391877 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:21.391960 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.129:40951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 16:43:21.428506 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 20038
I20260812 16:43:21.445668 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 19683
I20260812 16:43:21.446811 19744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:21.557639 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 19683
2026-08-12T16:43:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2817 ms)
[----------] 6 tests from EncryptionPolicyTest (21752 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:21 dist-test-slave-f7th krb5kdc[20223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:21 dist-test-slave-f7th krb5kdc[20223](info): set up 2 sockets
Aug 12 16:43:21 dist-test-slave-f7th krb5kdc[20223](info): commencing operation
krb5kdc: starting...
W20260812 16:43:22.612145 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.007s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:22 dist-test-slave-f7th krb5kdc[20223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:22.885802 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:23.225236 20237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:23.225538 20237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:23.225678 20237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:23.233124 20237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:23.233250 20237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:23.233310 20237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:23.233395 20237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:23.233443 20237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:23.243639 20237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:23.244997 20237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:23.246366 20237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:23.258173 20243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:23.258198 20242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:23.258245 20245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:23.259178 20237 server_base.cc:1034] running on GCE node
I20260812 16:43:23.260035 20237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:23.261701 20237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:23.262938 20237 hybrid_clock.cc:648] HybridClock initialized: now 1786553003262886 us; error 61 us; skew 500 ppm
I20260812 16:43:23.263291 20237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:23 dist-test-slave-f7th krb5kdc[20223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:23.266598 20237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:23.268338 20237 webserver.cc:460] Webserver started at http://127.0.0.1:36459/ using document root <none> and password file <none>
I20260812 16:43:23.269100 20237 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:23.269228 20237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:23.269563 20237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:23.272905 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "509ddf80032741469860bfc5ca6e38dd"
format_stamp: "Formatted at 2026-08-12 16:43:23 on dist-test-slave-f7th"
server_key: "1c0fd4e955397175b3e0d5c35d9a5044"
server_key_iv: "825bfaefd84ea5fad49ff45c9c8f3100"
server_key_version: "encryptionkey@0"
I20260812 16:43:23.273782 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "509ddf80032741469860bfc5ca6e38dd"
format_stamp: "Formatted at 2026-08-12 16:43:23 on dist-test-slave-f7th"
server_key: "1c0fd4e955397175b3e0d5c35d9a5044"
server_key_iv: "825bfaefd84ea5fad49ff45c9c8f3100"
server_key_version: "encryptionkey@0"
I20260812 16:43:23.279479 20237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:43:23.283406 20252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:23.285050 20237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:23.285246 20237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "509ddf80032741469860bfc5ca6e38dd"
format_stamp: "Formatted at 2026-08-12 16:43:23 on dist-test-slave-f7th"
server_key: "1c0fd4e955397175b3e0d5c35d9a5044"
server_key_iv: "825bfaefd84ea5fad49ff45c9c8f3100"
server_key_version: "encryptionkey@0"
I20260812 16:43:23.285449 20237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:23.322866 20237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:23.327407 20237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:23.327718 20237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:23.345710 20237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38513
I20260812 16:43:23.345741 20303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38513 every 8 connection(s)
I20260812 16:43:23.347106 20237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:23.351002 20304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:23.352620 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 20237
I20260812 16:43:23.352806 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:23.353433 31990 external_mini_cluster.cc:1428] Setting key 3625fec37f135b5f99caffe977b07a6e
I20260812 16:43:23.366137 20304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd: Bootstrap starting.
Aug 12 16:43:23 dist-test-slave-f7th krb5kdc[20223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 16:43:23.369758 20304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:23.370878 20304 log.cc:824] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:23.373759 20304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd: No bootstrap required, opened a new log
I20260812 16:43:23.376931 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:23.355396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43033 (local address 127.0.0.1:38513)
0812 16:43:23.356207 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:23.356219 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:23.356264 (+ 45us) server_negotiation.cc:407] Connection header received
0812 16:43:23.357822 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:23.357849 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:23.358267 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:23.358592 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:23.360514 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:23.361996 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:23.363937 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:23.364646 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:23.364782 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:23.369577 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:23.369592 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:23.369607 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:23.369651 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:23.372695 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:23.373502 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:23.373508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:23.373515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:23.373679 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:23.374174 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:23.374182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:23.374184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:23.374711 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:23.375025 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:23.375580 (+ 555us) server_negotiation.cc:299] Negotiation successful
0812 16:43:23.375897 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":352,"threads_started":1}
I20260812 16:43:23.380878 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:23.381798 20304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ddf80032741469860bfc5ca6e38dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } }
I20260812 16:43:23.382154 20304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:23.382247 20304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 509ddf80032741469860bfc5ca6e38dd, State: Initialized, Role: FOLLOWER
I20260812 16:43:23.382804 20304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ddf80032741469860bfc5ca6e38dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } }
I20260812 16:43:23.383044 20304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:23.383168 20304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:23.383327 20304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:23.386036 20304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ddf80032741469860bfc5ca6e38dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } }
I20260812 16:43:23.386646 20304 leader_election.cc:302] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 509ddf80032741469860bfc5ca6e38dd; no voters:
I20260812 16:43:23.387300 20304 leader_election.cc:288] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:23.387458 20316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:23.388140 20316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [term 1 LEADER]: Becoming Leader. State: Replica: 509ddf80032741469860bfc5ca6e38dd, State: Running, Role: LEADER
I20260812 16:43:23.388909 20316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ddf80032741469860bfc5ca6e38dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } }
I20260812 16:43:23.390460 20304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:23.391788 20318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "509ddf80032741469860bfc5ca6e38dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ddf80032741469860bfc5ca6e38dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } } }
I20260812 16:43:23.392185 20318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:23.397265 20317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 509ddf80032741469860bfc5ca6e38dd. Latest consensus state: current_term: 1 leader_uuid: "509ddf80032741469860bfc5ca6e38dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ddf80032741469860bfc5ca6e38dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38513 } } }
I20260812 16:43:23.398329 20317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:23.402184 20321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:23.404551 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:23.385704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59811 (local address 127.0.0.1:38513)
0812 16:43:23.385873 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:23.385878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:23.386265 (+ 387us) server_negotiation.cc:407] Connection header received
0812 16:43:23.386519 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:23.386525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:23.386603 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:23.386741 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:23.388608 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:23.389806 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:23.391746 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:23.392515 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:23.392648 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:23.396382 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:23.396390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:23.396396 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:23.396447 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:23.400427 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:23.401650 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:23.401656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:23.401661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:23.401850 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:23.402477 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:23.402483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:23.402485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:23.403103 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:23.403476 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:23.403901 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 16:43:23.404123 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 16:43:23.408581 20321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:23.419342 20321 catalog_manager.cc:1347] Generated new cluster ID: 155239f775474700813f66ae64afa212
I20260812 16:43:23.419484 20321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:23.426898 20321 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:23.428488 20321 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:23.436844 20321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 509ddf80032741469860bfc5ca6e38dd: Generated new TSK 0
I20260812 16:43:23.437718 20321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:23.449878 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20237
I20260812 16:43:23.451157 20299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:23.541442 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 20237
W20260812 16:43:23.556486 20221 connection.cc:535] client connection to 127.0.0.1:38513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38513): Connection reset by peer (error 104)
2026-08-12T16:43:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:23 dist-test-slave-f7th krb5kdc[20338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:23 dist-test-slave-f7th krb5kdc[20338](info): set up 2 sockets
Aug 12 16:43:23 dist-test-slave-f7th krb5kdc[20338](info): commencing operation
krb5kdc: starting...
W20260812 16:43:24.593017 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:24 dist-test-slave-f7th krb5kdc[20338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:24.859365 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:25.197136 20351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:25.197458 20351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:25.197602 20351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:25.205102 20351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:25.205240 20351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:25.205322 20351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:25.205403 20351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:25.205467 20351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:25.215654 20351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:25.217017 20351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:25.218354 20351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:25.231106 20356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:25.231140 20359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:25.231077 20357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:25.232267 20351 server_base.cc:1034] running on GCE node
I20260812 16:43:25.233179 20351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:25.234680 20351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:25.235894 20351 hybrid_clock.cc:648] HybridClock initialized: now 1786553005235830 us; error 55 us; skew 500 ppm
I20260812 16:43:25.236260 20351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:25 dist-test-slave-f7th krb5kdc[20338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:25.239701 20351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:25.241245 20351 webserver.cc:460] Webserver started at http://127.0.0.1:41179/ using document root <none> and password file <none>
I20260812 16:43:25.241922 20351 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:25.242031 20351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:25.242302 20351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:25.245488 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "dec26d079fb84c2cba3ecda89b905338"
format_stamp: "Formatted at 2026-08-12 16:43:25 on dist-test-slave-f7th"
server_key: "fb383f99c61ebbeb825d255fff6fb20b"
server_key_iv: "977c3115572a32f06bc0d6ef28a705df"
server_key_version: "encryptionkey@0"
I20260812 16:43:25.246264 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "dec26d079fb84c2cba3ecda89b905338"
format_stamp: "Formatted at 2026-08-12 16:43:25 on dist-test-slave-f7th"
server_key: "fb383f99c61ebbeb825d255fff6fb20b"
server_key_iv: "977c3115572a32f06bc0d6ef28a705df"
server_key_version: "encryptionkey@0"
I20260812 16:43:25.251808 20351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:43:25.255751 20366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:25.257385 20351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:43:25.257571 20351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "dec26d079fb84c2cba3ecda89b905338"
format_stamp: "Formatted at 2026-08-12 16:43:25 on dist-test-slave-f7th"
server_key: "fb383f99c61ebbeb825d255fff6fb20b"
server_key_iv: "977c3115572a32f06bc0d6ef28a705df"
server_key_version: "encryptionkey@0"
I20260812 16:43:25.257809 20351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:25.274561 20351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:25.279150 20351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:25.279443 20351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:25.298956 20351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35037
I20260812 16:43:25.299062 20418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35037 every 8 connection(s)
I20260812 16:43:25.300464 20351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:25.304451 20419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:25.305337 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 20351
I20260812 16:43:25.305485 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:25.305929 31990 external_mini_cluster.cc:1428] Setting key d11215b3ec3491c1a8770f75d5459821
I20260812 16:43:25.319772 20419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338: Bootstrap starting.
I20260812 16:43:25.320858 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:25.307880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60771 (local address 127.0.0.1:35037)
0812 16:43:25.308725 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:25.308741 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:25.308793 (+ 52us) server_negotiation.cc:407] Connection header received
0812 16:43:25.310515 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:25.310551 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:25.311122 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:25.311589 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:25.312191 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:25.314118 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:25.315499 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:25.316617 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:25.316808 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:25.317997 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:25.318008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:25.318028 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:25.318740 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:25.318847 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:25.318878 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:25.319039 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:25.319290 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 16:43:25.319475 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":361,"threads_started":1}
I20260812 16:43:25.323482 20419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:25.324661 20419 log.cc:824] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:25.325116 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:25.328011 20419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338: No bootstrap required, opened a new log
I20260812 16:43:25.337509 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:25.330316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55609 (local address 127.0.0.1:35037)
0812 16:43:25.330494 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:25.330501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:25.330819 (+ 318us) server_negotiation.cc:407] Connection header received
0812 16:43:25.330991 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:25.330996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:25.331078 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:25.331210 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:25.331733 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:25.332913 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:25.334574 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:25.335258 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:25.335406 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:25.336675 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:25.336682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:25.336686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:25.336707 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:25.336759 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:25.336779 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:25.336894 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:25.337124 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 16:43:25.337180 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 16:43:25.337709 20419 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec26d079fb84c2cba3ecda89b905338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } }
I20260812 16:43:25.338214 20419 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:25.338331 20419 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec26d079fb84c2cba3ecda89b905338, State: Initialized, Role: FOLLOWER
I20260812 16:43:25.339068 20419 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec26d079fb84c2cba3ecda89b905338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } }
I20260812 16:43:25.339296 20419 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:25.339430 20419 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:25.339587 20419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:25.341845 20419 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec26d079fb84c2cba3ecda89b905338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } }
I20260812 16:43:25.342334 20419 leader_election.cc:302] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec26d079fb84c2cba3ecda89b905338; no voters:
I20260812 16:43:25.342893 20419 leader_election.cc:288] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:25.343034 20429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:25.344494 20429 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [term 1 LEADER]: Becoming Leader. State: Replica: dec26d079fb84c2cba3ecda89b905338, State: Running, Role: LEADER
I20260812 16:43:25.345098 20419 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:25.345090 20429 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec26d079fb84c2cba3ecda89b905338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } }
I20260812 16:43:25.348448 20431 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec26d079fb84c2cba3ecda89b905338. Latest consensus state: current_term: 1 leader_uuid: "dec26d079fb84c2cba3ecda89b905338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec26d079fb84c2cba3ecda89b905338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } } }
I20260812 16:43:25.348759 20431 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:25.349195 20430 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec26d079fb84c2cba3ecda89b905338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec26d079fb84c2cba3ecda89b905338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35037 } } }
I20260812 16:43:25.349521 20430 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:25.349725 20439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:25.356361 20439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:25.365502 20439 catalog_manager.cc:1347] Generated new cluster ID: 2e01a20b5bff451b90ac861c0d9bdf7a
I20260812 16:43:25.365612 20439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:25.399931 20439 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:25.401373 20439 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:25.410993 20439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec26d079fb84c2cba3ecda89b905338: Generated new TSK 0
I20260812 16:43:25.412133 20439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:25.422648 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20351
W20260812 16:43:25.423784 20414 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60771
W20260812 16:43:25.424623 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:25.424811 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 20351
W20260812 16:43:25.439808 20336 connection.cc:535] client connection to 127.0.0.1:35037 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35037): Connection reset by peer (error 104)
2026-08-12T16:43:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:25 dist-test-slave-f7th krb5kdc[20452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:25 dist-test-slave-f7th krb5kdc[20452](info): set up 2 sockets
Aug 12 16:43:25 dist-test-slave-f7th krb5kdc[20452](info): commencing operation
krb5kdc: starting...
W20260812 16:43:26.479485 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:26 dist-test-slave-f7th krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:26.756283 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:27.101235 20465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:27.101661 20465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:27.101918 20465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:27.110013 20465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:27.110150 20465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:27.110231 20465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:27.110317 20465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:27.110378 20465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:27.121142 20465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:27.122540 20465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:27.124056 20465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:27.136356 20470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:27.136435 20471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:27.136529 20473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:27.137809 20465 server_base.cc:1034] running on GCE node
I20260812 16:43:27.138664 20465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:27.140373 20465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:27.141685 20465 hybrid_clock.cc:648] HybridClock initialized: now 1786553007141624 us; error 43 us; skew 500 ppm
I20260812 16:43:27.142042 20465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:27 dist-test-slave-f7th krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:27.150719 20465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:27.152562 20465 webserver.cc:460] Webserver started at http://127.0.0.1:33065/ using document root <none> and password file <none>
I20260812 16:43:27.153344 20465 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:27.153486 20465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:27.153803 20465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:27.157122 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "c4ea170712e4450e9da7c524dffc67ef"
format_stamp: "Formatted at 2026-08-12 16:43:27 on dist-test-slave-f7th"
server_key: "f3668fca93666f504fd26720157a0773"
server_key_iv: "c3a010dcc403d4b6c7799739c2051f30"
server_key_version: "encryptionkey@0"
I20260812 16:43:27.157982 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "c4ea170712e4450e9da7c524dffc67ef"
format_stamp: "Formatted at 2026-08-12 16:43:27 on dist-test-slave-f7th"
server_key: "f3668fca93666f504fd26720157a0773"
server_key_iv: "c3a010dcc403d4b6c7799739c2051f30"
server_key_version: "encryptionkey@0"
I20260812 16:43:27.163539 20465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:43:27.167212 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:27.168656 20465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:27.168834 20465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "c4ea170712e4450e9da7c524dffc67ef"
format_stamp: "Formatted at 2026-08-12 16:43:27 on dist-test-slave-f7th"
server_key: "f3668fca93666f504fd26720157a0773"
server_key_iv: "c3a010dcc403d4b6c7799739c2051f30"
server_key_version: "encryptionkey@0"
I20260812 16:43:27.169036 20465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:27.174957 20465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:27.179113 20465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:27.179397 20465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:27.198040 20465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32871
I20260812 16:43:27.198079 20532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32871 every 8 connection(s)
I20260812 16:43:27.199471 20465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:27.203361 20534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:27.206723 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 20465
I20260812 16:43:27.206914 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:27.207341 31990 external_mini_cluster.cc:1428] Setting key d94ca5e0b94c457a65f84d0a3f502d59
I20260812 16:43:27.215729 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:27.209313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35223 (local address 127.0.0.1:32871)
0812 16:43:27.210206 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:27.210219 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:27.210265 (+ 46us) server_negotiation.cc:407] Connection header received
0812 16:43:27.212058 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:27.212085 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:27.212501 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:27.212915 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:27.213945 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:27.213952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:27.213966 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:27.213982 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:27.214046 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:27.214066 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:27.214286 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:27.214428 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 16:43:27.214722 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":393,"threads_started":1}
I20260812 16:43:27.216109 20534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef: Bootstrap starting.
I20260812 16:43:27.219990 20534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:27.220022 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:27.221068 20534 log.cc:824] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:27.224408 20534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef: No bootstrap required, opened a new log
I20260812 16:43:27.227867 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:27.224681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40333 (local address 127.0.0.1:32871)
0812 16:43:27.224869 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:27.224876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:27.225452 (+ 576us) server_negotiation.cc:407] Connection header received
0812 16:43:27.225863 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:27.225868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:27.225942 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:27.226060 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:27.227059 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:27.227063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:27.227067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:27.227083 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:27.227130 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:27.227149 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:27.227253 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:27.227427 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 16:43:27.227507 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:43:27.233079 20534 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ea170712e4450e9da7c524dffc67ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32871 } }
I20260812 16:43:27.233517 20534 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:27.233724 20534 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ea170712e4450e9da7c524dffc67ef, State: Initialized, Role: FOLLOWER
I20260812 16:43:27.234396 20534 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ea170712e4450e9da7c524dffc67ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32871 } }
I20260812 16:43:27.234656 20534 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:27.234793 20534 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:27.234948 20534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:27.237476 20534 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ea170712e4450e9da7c524dffc67ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32871 } }
I20260812 16:43:27.238016 20534 leader_election.cc:302] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4ea170712e4450e9da7c524dffc67ef; no voters:
I20260812 16:43:27.238720 20534 leader_election.cc:288] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:27.238811 20544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:27.239900 20544 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [term 1 LEADER]: Becoming Leader. State: Replica: c4ea170712e4450e9da7c524dffc67ef, State: Running, Role: LEADER
I20260812 16:43:27.240435 20544 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ea170712e4450e9da7c524dffc67ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32871 } }
I20260812 16:43:27.240877 20534 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:27.243402 20545 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4ea170712e4450e9da7c524dffc67ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ea170712e4450e9da7c524dffc67ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32871 } } }
I20260812 16:43:27.244131 20545 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:27.243427 20546 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ea170712e4450e9da7c524dffc67ef. Latest consensus state: current_term: 1 leader_uuid: "c4ea170712e4450e9da7c524dffc67ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ea170712e4450e9da7c524dffc67ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32871 } } }
I20260812 16:43:27.244671 20546 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:27.246179 20552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:27.251298 20552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:27.262287 20552 catalog_manager.cc:1347] Generated new cluster ID: c6bfb7fe524145a1b704a3955ec4a7b0
I20260812 16:43:27.262396 20552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:27.301939 20552 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:27.302418 20552 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:27.311174 20552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4ea170712e4450e9da7c524dffc67ef: Generated new TSK 0
I20260812 16:43:27.312291 20552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:27.327980 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20465
W20260812 16:43:27.329586 20528 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35223
W20260812 16:43:27.330427 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20465: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:27.330597 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 20465
2026-08-12T16:43:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:27 dist-test-slave-f7th krb5kdc[20565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:27 dist-test-slave-f7th krb5kdc[20565](info): set up 2 sockets
Aug 12 16:43:27 dist-test-slave-f7th krb5kdc[20565](info): commencing operation
krb5kdc: starting...
W20260812 16:43:28.395558 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:28 dist-test-slave-f7th krb5kdc[20565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:28.666966 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:29.004019 20579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:29.004351 20579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:29.004469 20579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:29.011986 20579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:29.012116 20579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:29.012193 20579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:29.012270 20579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:29.012329 20579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:29.022593 20579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:29.024010 20579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:29.025352 20579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:29.037287 20584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:29.037320 20585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:29.037380 20587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:29.038187 20579 server_base.cc:1034] running on GCE node
I20260812 16:43:29.038992 20579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:29.040512 20579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:29.041716 20579 hybrid_clock.cc:648] HybridClock initialized: now 1786553009041673 us; error 33 us; skew 500 ppm
I20260812 16:43:29.042073 20579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:29 dist-test-slave-f7th krb5kdc[20565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:29.045279 20579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:29.046913 20579 webserver.cc:460] Webserver started at http://127.0.0.1:33765/ using document root <none> and password file <none>
I20260812 16:43:29.047654 20579 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:29.047784 20579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:29.048089 20579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:29.051200 20579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "831c7706b89c43e2a0b2df062d126d4a"
format_stamp: "Formatted at 2026-08-12 16:43:29 on dist-test-slave-f7th"
server_key: "cdc42982a2e4f723edc2b2b5fe789ae2"
server_key_iv: "2a17b274ce32ebfbdb2a664ec961aadc"
server_key_version: "encryptionkey@0"
I20260812 16:43:29.052028 20579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "831c7706b89c43e2a0b2df062d126d4a"
format_stamp: "Formatted at 2026-08-12 16:43:29 on dist-test-slave-f7th"
server_key: "cdc42982a2e4f723edc2b2b5fe789ae2"
server_key_iv: "2a17b274ce32ebfbdb2a664ec961aadc"
server_key_version: "encryptionkey@0"
I20260812 16:43:29.057389 20579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260812 16:43:29.060971 20595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:29.062453 20579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 16:43:29.062642 20579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "831c7706b89c43e2a0b2df062d126d4a"
format_stamp: "Formatted at 2026-08-12 16:43:29 on dist-test-slave-f7th"
server_key: "cdc42982a2e4f723edc2b2b5fe789ae2"
server_key_iv: "2a17b274ce32ebfbdb2a664ec961aadc"
server_key_version: "encryptionkey@0"
I20260812 16:43:29.062826 20579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:29.081207 20579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:29.085855 20579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:29.086194 20579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:29.105391 20579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34995
I20260812 16:43:29.105469 20648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34995 every 8 connection(s)
I20260812 16:43:29.106809 20579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:29.110706 20649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:29.113659 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 20579
I20260812 16:43:29.113835 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:29.114389 31990 external_mini_cluster.cc:1428] Setting key e7ee03a888cedd09c7e8989fd452b0c8
I20260812 16:43:29.122943 20649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a: Bootstrap starting.
I20260812 16:43:29.126363 20649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:29.127458 20649 log.cc:824] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a: Log is configured to *not* fsync() on all Append() calls
Aug 12 16:43:29 dist-test-slave-f7th krb5kdc[20565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 16:43:29.130501 20649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a: No bootstrap required, opened a new log
I20260812 16:43:29.137622 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:29.116331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42021 (local address 127.0.0.1:34995)
0812 16:43:29.117174 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:29.117187 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:29.117230 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:43:29.119035 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:29.119061 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:29.119494 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:29.119932 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:29.121412 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:29.122742 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:29.124147 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:29.124866 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:29.125064 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:29.129340 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:29.129349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:29.129365 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:29.129405 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:29.132751 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:29.133533 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:29.133542 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:29.133549 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:29.133745 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:29.134153 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:29.134158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:29.134160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:29.134771 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:29.135088 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:29.135585 (+ 497us) server_negotiation.cc:299] Negotiation successful
0812 16:43:29.136285 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":372,"threads_started":1}
I20260812 16:43:29.139638 20649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831c7706b89c43e2a0b2df062d126d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34995 } }
I20260812 16:43:29.140142 20649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:29.140261 20649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831c7706b89c43e2a0b2df062d126d4a, State: Initialized, Role: FOLLOWER
I20260812 16:43:29.140995 20649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831c7706b89c43e2a0b2df062d126d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34995 } }
I20260812 16:43:29.141276 20649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:29.141453 20649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:29.141736 20649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:29.142839 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:29.144896 20649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831c7706b89c43e2a0b2df062d126d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34995 } }
I20260812 16:43:29.145568 20649 leader_election.cc:302] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 831c7706b89c43e2a0b2df062d126d4a; no voters:
I20260812 16:43:29.146654 20649 leader_election.cc:288] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:29.146750 20654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:29.150435 20649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:29.153632 20654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [term 1 LEADER]: Becoming Leader. State: Replica: 831c7706b89c43e2a0b2df062d126d4a, State: Running, Role: LEADER
I20260812 16:43:29.154228 20654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831c7706b89c43e2a0b2df062d126d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34995 } }
I20260812 16:43:29.157641 20657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "831c7706b89c43e2a0b2df062d126d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831c7706b89c43e2a0b2df062d126d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34995 } } }
I20260812 16:43:29.158519 20659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 831c7706b89c43e2a0b2df062d126d4a. Latest consensus state: current_term: 1 leader_uuid: "831c7706b89c43e2a0b2df062d126d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831c7706b89c43e2a0b2df062d126d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34995 } } }
I20260812 16:43:29.158821 20659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:29.159337 20657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:29.170149 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:29.148927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36835 (local address 127.0.0.1:34995)
0812 16:43:29.149137 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:29.149144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:29.150188 (+ 1044us) server_negotiation.cc:407] Connection header received
0812 16:43:29.150419 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:29.150424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:29.150520 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:29.150637 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:29.152911 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:29.154219 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:29.156446 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:29.157145 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:29.157257 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:29.160709 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:29.160716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:29.160722 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:29.160766 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:29.165315 (+ 4549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:29.166487 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:29.166494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:29.166498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:29.166702 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:29.167311 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:29.167317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:29.167321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:29.167823 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:29.168076 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:29.169469 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0812 16:43:29.169742 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":79}
I20260812 16:43:29.174216 20670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:29.179591 20670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:29.189875 20670 catalog_manager.cc:1347] Generated new cluster ID: dde0813907454f34b3fc0b67328be378
I20260812 16:43:29.190017 20670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:29.212512 20670 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:29.214000 20670 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:29.220554 20670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 831c7706b89c43e2a0b2df062d126d4a: Generated new TSK 0
I20260812 16:43:29.221974 20670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:29.227762 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20579
I20260812 16:43:29.228998 20643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:29.324836 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 20579
2026-08-12T16:43:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1994 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:29 dist-test-slave-f7th krb5kdc[20682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:29 dist-test-slave-f7th krb5kdc[20682](info): set up 2 sockets
Aug 12 16:43:29 dist-test-slave-f7th krb5kdc[20682](info): commencing operation
krb5kdc: starting...
W20260812 16:43:30.383563 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:30 dist-test-slave-f7th krb5kdc[20682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:30.655232 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:30.994849 20696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:30.995167 20696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:30.995292 20696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:31.002804 20696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:31.002948 20696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:31.002995 20696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:31.003058 20696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:31.003120 20696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:31.013314 20696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:31.015027 20696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:31.016430 20696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:31.028183 20704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:31.028273 20701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:31.028266 20702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:31.029166 20696 server_base.cc:1034] running on GCE node
I20260812 16:43:31.030019 20696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:31.031561 20696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:31.032814 20696 hybrid_clock.cc:648] HybridClock initialized: now 1786553011032756 us; error 45 us; skew 500 ppm
I20260812 16:43:31.033183 20696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:31 dist-test-slave-f7th krb5kdc[20682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:31.036644 20696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:31.038393 20696 webserver.cc:460] Webserver started at http://127.0.0.1:44423/ using document root <none> and password file <none>
I20260812 16:43:31.039124 20696 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:31.039244 20696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:31.039568 20696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:31.042871 20696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "7ef6847ee28d48918a421fef96c9bff8"
format_stamp: "Formatted at 2026-08-12 16:43:31 on dist-test-slave-f7th"
server_key: "e5add30142de908978cd5a759e2b1696"
server_key_iv: "f9f6c7ee25508233598cbe67bdf98acc"
server_key_version: "encryptionkey@0"
I20260812 16:43:31.043753 20696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "7ef6847ee28d48918a421fef96c9bff8"
format_stamp: "Formatted at 2026-08-12 16:43:31 on dist-test-slave-f7th"
server_key: "e5add30142de908978cd5a759e2b1696"
server_key_iv: "f9f6c7ee25508233598cbe67bdf98acc"
server_key_version: "encryptionkey@0"
I20260812 16:43:31.049460 20696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:43:31.053514 20712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:31.055002 20696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:43:31.055223 20696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "7ef6847ee28d48918a421fef96c9bff8"
format_stamp: "Formatted at 2026-08-12 16:43:31 on dist-test-slave-f7th"
server_key: "e5add30142de908978cd5a759e2b1696"
server_key_iv: "f9f6c7ee25508233598cbe67bdf98acc"
server_key_version: "encryptionkey@0"
I20260812 16:43:31.055482 20696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:31.083201 20696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:31.087785 20696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:31.088073 20696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:31.110875 20696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43687
I20260812 16:43:31.110914 20764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43687 every 8 connection(s)
I20260812 16:43:31.112701 20696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:31.117410 20765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:31.122763 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 20696
I20260812 16:43:31.122941 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:31.123466 31990 external_mini_cluster.cc:1428] Setting key cf87f92b68f4baa352e7705fb4013cbc
I20260812 16:43:31.129681 20765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8: Bootstrap starting.
I20260812 16:43:31.133919 20765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:31.135143 20765 log.cc:824] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:31.138940 20765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8: No bootstrap required, opened a new log
I20260812 16:43:31.143208 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:31.126075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49443 (local address 127.0.0.1:43687)
0812 16:43:31.127001 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:31.127018 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:31.127075 (+ 57us) server_negotiation.cc:407] Connection header received
0812 16:43:31.129004 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:31.129031 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:31.129475 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:31.129855 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:31.130491 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:31.132518 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:31.134407 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:31.135583 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:31.135864 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:31.140151 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:31.140160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:31.140177 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:31.140936 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:31.141026 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:31.141060 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:31.141403 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:31.141742 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 16:43:31.142109 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":932,"threads_started":1}
I20260812 16:43:31.148123 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:31.150933 20765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef6847ee28d48918a421fef96c9bff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260812 16:43:31.151587 20765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:31.151749 20765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef6847ee28d48918a421fef96c9bff8, State: Initialized, Role: FOLLOWER
I20260812 16:43:31.152493 20765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef6847ee28d48918a421fef96c9bff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260812 16:43:31.152855 20765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:31.153007 20765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:31.153198 20765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:31.156265 20765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef6847ee28d48918a421fef96c9bff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260812 16:43:31.156919 20765 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ef6847ee28d48918a421fef96c9bff8; no voters:
I20260812 16:43:31.157682 20765 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:31.157805 20775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:31.159016 20775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7ef6847ee28d48918a421fef96c9bff8, State: Running, Role: LEADER
I20260812 16:43:31.159767 20775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef6847ee28d48918a421fef96c9bff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260812 16:43:31.160370 20765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:31.161116 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:31.153517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48123 (local address 127.0.0.1:43687)
0812 16:43:31.153756 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:31.153762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:31.154049 (+ 287us) server_negotiation.cc:407] Connection header received
0812 16:43:31.154254 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:31.154259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:31.154341 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:31.154428 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:31.155032 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:31.155893 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:31.157746 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:31.158517 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:31.159401 (+ 884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:31.160281 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:31.160288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:31.160293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:31.160316 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:31.160372 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:31.160396 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:31.160585 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:31.160670 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 16:43:31.160766 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 16:43:31.162863 20776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ef6847ee28d48918a421fef96c9bff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef6847ee28d48918a421fef96c9bff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260812 16:43:31.163347 20776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:31.163906 20777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef6847ee28d48918a421fef96c9bff8. Latest consensus state: current_term: 1 leader_uuid: "7ef6847ee28d48918a421fef96c9bff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef6847ee28d48918a421fef96c9bff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260812 16:43:31.164304 20777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:31.168175 20782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:31.173228 20782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:31.182520 20782 catalog_manager.cc:1347] Generated new cluster ID: bd65d1de498b460098095e7829ab3889
I20260812 16:43:31.182657 20782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:31.206137 20782 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:31.207484 20782 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:31.215762 20782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ef6847ee28d48918a421fef96c9bff8: Generated new TSK 0
I20260812 16:43:31.216629 20782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:31.230881 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20696
W20260812 16:43:31.232172 20760 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49443
W20260812 16:43:31.232878 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20696: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:31.233038 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 20696
2026-08-12T16:43:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1906 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:31 dist-test-slave-f7th krb5kdc[20796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:31 dist-test-slave-f7th krb5kdc[20796](info): set up 2 sockets
Aug 12 16:43:31 dist-test-slave-f7th krb5kdc[20796](info): commencing operation
krb5kdc: starting...
W20260812 16:43:32.275559 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:32 dist-test-slave-f7th krb5kdc[20796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:32.557027 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:32.891322 20810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:32.891695 20810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:32.891845 20810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:32.899070 20810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:32.899202 20810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:32.899284 20810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:32.899363 20810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:32.899423 20810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:32.909878 20810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:32.911262 20810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:32.912640 20810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:32.925273 20818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:32.925339 20816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:32.925303 20815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:32.925915 20810 server_base.cc:1034] running on GCE node
I20260812 16:43:32.926841 20810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:32.928395 20810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:32.929615 20810 hybrid_clock.cc:648] HybridClock initialized: now 1786553012929570 us; error 53 us; skew 500 ppm
I20260812 16:43:32.929975 20810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:32 dist-test-slave-f7th krb5kdc[20796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:32.936901 20810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:32.938654 20810 webserver.cc:460] Webserver started at http://127.0.0.1:41659/ using document root <none> and password file <none>
I20260812 16:43:32.939383 20810 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:32.939519 20810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:32.939850 20810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:32.943157 20810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "cae73560cfa74b6792c27d6b62b2d187"
format_stamp: "Formatted at 2026-08-12 16:43:32 on dist-test-slave-f7th"
server_key: "d2528a4e28a62089f3f60b940c1f0924"
server_key_iv: "f195154f78c061f3e96b155d81f1be06"
server_key_version: "encryptionkey@0"
I20260812 16:43:32.944000 20810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "cae73560cfa74b6792c27d6b62b2d187"
format_stamp: "Formatted at 2026-08-12 16:43:32 on dist-test-slave-f7th"
server_key: "d2528a4e28a62089f3f60b940c1f0924"
server_key_iv: "f195154f78c061f3e96b155d81f1be06"
server_key_version: "encryptionkey@0"
I20260812 16:43:32.949539 20810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:43:32.953531 20825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:32.955001 20810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:43:32.955194 20810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "cae73560cfa74b6792c27d6b62b2d187"
format_stamp: "Formatted at 2026-08-12 16:43:32 on dist-test-slave-f7th"
server_key: "d2528a4e28a62089f3f60b940c1f0924"
server_key_iv: "f195154f78c061f3e96b155d81f1be06"
server_key_version: "encryptionkey@0"
I20260812 16:43:32.955389 20810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:32.961397 20810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:32.965478 20810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:32.965764 20810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:32.984366 20810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35019
I20260812 16:43:32.984439 20877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35019 every 8 connection(s)
I20260812 16:43:32.986034 20810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:32.990309 20878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:32.993700 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 20810
I20260812 16:43:32.993911 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:32.994504 31990 external_mini_cluster.cc:1428] Setting key f878a064028c0aa3d9dc21be2635230e
I20260812 16:43:33.004020 20878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187: Bootstrap starting.
I20260812 16:43:33.004159 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:32.996675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53095 (local address 127.0.0.1:35019)
0812 16:43:32.997481 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:32.997493 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:32.997538 (+ 45us) server_negotiation.cc:407] Connection header received
0812 16:43:33.000033 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:33.000086 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:33.000879 (+ 793us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:33.001361 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:33.002411 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:33.002418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:33.002432 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:33.002448 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:33.002519 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:33.002541 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:33.002767 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:33.002905 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 16:43:33.003099 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":360,"threads_started":1}
I20260812 16:43:33.007391 20878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:33.007972 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:33.008584 20878 log.cc:824] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:33.011660 20878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187: No bootstrap required, opened a new log
I20260812 16:43:33.014999 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:33.012341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49489 (local address 127.0.0.1:35019)
0812 16:43:33.012499 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:33.012503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:33.012957 (+ 454us) server_negotiation.cc:407] Connection header received
0812 16:43:33.013135 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:33.013139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:33.013190 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:33.013273 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:33.014317 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:33.014322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:33.014325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:33.014345 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:33.014386 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:33.014405 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:33.014526 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:33.014651 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 16:43:33.014722 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 16:43:33.019984 20878 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae73560cfa74b6792c27d6b62b2d187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } }
I20260812 16:43:33.020408 20878 raft_consensus.cc:374] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:33.020505 20878 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae73560cfa74b6792c27d6b62b2d187, State: Initialized, Role: FOLLOWER
I20260812 16:43:33.021212 20878 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae73560cfa74b6792c27d6b62b2d187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } }
I20260812 16:43:33.021484 20878 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:33.021631 20878 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:33.021813 20878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:33.024215 20878 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae73560cfa74b6792c27d6b62b2d187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } }
I20260812 16:43:33.024753 20878 leader_election.cc:302] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae73560cfa74b6792c27d6b62b2d187; no voters:
I20260812 16:43:33.025313 20878 leader_election.cc:288] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:33.025460 20889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:33.026096 20889 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [term 1 LEADER]: Becoming Leader. State: Replica: cae73560cfa74b6792c27d6b62b2d187, State: Running, Role: LEADER
I20260812 16:43:33.026755 20889 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae73560cfa74b6792c27d6b62b2d187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } }
I20260812 16:43:33.027624 20878 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:33.029659 20890 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cae73560cfa74b6792c27d6b62b2d187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae73560cfa74b6792c27d6b62b2d187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } } }
I20260812 16:43:33.030184 20890 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:33.029628 20891 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae73560cfa74b6792c27d6b62b2d187. Latest consensus state: current_term: 1 leader_uuid: "cae73560cfa74b6792c27d6b62b2d187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae73560cfa74b6792c27d6b62b2d187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } } }
I20260812 16:43:33.030741 20891 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:33.031165 20897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:33.036590 20897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:33.044956 20897 catalog_manager.cc:1347] Generated new cluster ID: 477225c1dedc43978fdfd98e723d0b5e
I20260812 16:43:33.045085 20897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:33.058964 20897 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:33.059360 20897 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:33.070122 20897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae73560cfa74b6792c27d6b62b2d187: Generated new TSK 0
I20260812 16:43:33.071245 20897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:33.084946 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20810
W20260812 16:43:33.086117 20873 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53095
W20260812 16:43:33.086866 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20810: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:33.087000 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 20810
2026-08-12T16:43:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1854 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:33 dist-test-slave-f7th krb5kdc[20911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:33 dist-test-slave-f7th krb5kdc[20911](info): set up 2 sockets
Aug 12 16:43:33 dist-test-slave-f7th krb5kdc[20911](info): commencing operation
krb5kdc: starting...
W20260812 16:43:34.144002 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:34 dist-test-slave-f7th krb5kdc[20911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:34.427858 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:34.770145 20924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:34.770467 20924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:34.770603 20924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:34.778146 20924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:34.778277 20924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:34.778350 20924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:34.778434 20924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:34.778493 20924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:34.789227 20924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:34.790552 20924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:34.791901 20924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:34.803558 20932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:34.803660 20930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:34.803565 20929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:34.804657 20924 server_base.cc:1034] running on GCE node
I20260812 16:43:34.805440 20924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:34.806867 20924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:34.808072 20924 hybrid_clock.cc:648] HybridClock initialized: now 1786553014808005 us; error 50 us; skew 500 ppm
I20260812 16:43:34.808414 20924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:34 dist-test-slave-f7th krb5kdc[20911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:34.811391 20924 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:34.812939 20924 webserver.cc:460] Webserver started at http://127.0.0.1:39937/ using document root <none> and password file <none>
I20260812 16:43:34.813627 20924 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:34.813815 20924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:34.814179 20924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:34.817286 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "6f6f85f573ff431889577170d7814910"
format_stamp: "Formatted at 2026-08-12 16:43:34 on dist-test-slave-f7th"
server_key: "6b339ffe86c50cf96f3e3fa8cf1e246f"
server_key_iv: "963b4000f49066cff871b67e2ae34bd8"
server_key_version: "encryptionkey@0"
I20260812 16:43:34.818089 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "6f6f85f573ff431889577170d7814910"
format_stamp: "Formatted at 2026-08-12 16:43:34 on dist-test-slave-f7th"
server_key: "6b339ffe86c50cf96f3e3fa8cf1e246f"
server_key_iv: "963b4000f49066cff871b67e2ae34bd8"
server_key_version: "encryptionkey@0"
I20260812 16:43:34.823647 20924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260812 16:43:34.827564 20939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:34.829109 20924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 16:43:34.829306 20924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "6f6f85f573ff431889577170d7814910"
format_stamp: "Formatted at 2026-08-12 16:43:34 on dist-test-slave-f7th"
server_key: "6b339ffe86c50cf96f3e3fa8cf1e246f"
server_key_iv: "963b4000f49066cff871b67e2ae34bd8"
server_key_version: "encryptionkey@0"
I20260812 16:43:34.829532 20924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:34.840914 20924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:34.845348 20924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:34.845630 20924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:34.863276 20924 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46097
I20260812 16:43:34.863297 20993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46097 every 8 connection(s)
I20260812 16:43:34.864713 20924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:34.868386 20994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:34.874982 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 20924
I20260812 16:43:34.875152 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:34.875579 31990 external_mini_cluster.cc:1428] Setting key 4119b5d4acef26d345141582e5340e45
I20260812 16:43:34.879328 20994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910: Bootstrap starting.
I20260812 16:43:34.883124 20994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:34.884254 20994 log.cc:824] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:34.887462 20994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910: No bootstrap required, opened a new log
Aug 12 16:43:34 dist-test-slave-f7th krb5kdc[20911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 16:43:34.896520 20994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f85f573ff431889577170d7814910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46097 } }
I20260812 16:43:34.896920 20994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:34.897085 20994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6f85f573ff431889577170d7814910, State: Initialized, Role: FOLLOWER
I20260812 16:43:34.897709 20994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f85f573ff431889577170d7814910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46097 } }
I20260812 16:43:34.898008 20994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:34.898144 20994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:34.898314 20994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:34.899169 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:34.877440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:34501 (local address 127.0.0.1:46097)
0812 16:43:34.878214 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:34.878226 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:34.878268 (+ 42us) server_negotiation.cc:407] Connection header received
0812 16:43:34.880017 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:34.880044 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:34.880456 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:34.880786 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:34.882622 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:34.884081 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:34.885593 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:34.886265 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:34.886406 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:34.891436 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:34.891446 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:34.891466 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:34.891512 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:34.894600 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:34.895577 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:34.895584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:34.895591 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:34.895815 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:34.896293 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:34.896297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:34.896299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:34.896832 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:34.897177 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:34.897695 (+ 518us) server_negotiation.cc:299] Negotiation successful
0812 16:43:34.898042 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":350,"threads_started":1}
I20260812 16:43:34.900892 20994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f85f573ff431889577170d7814910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46097 } }
I20260812 16:43:34.901408 20994 leader_election.cc:302] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f6f85f573ff431889577170d7814910; no voters:
I20260812 16:43:34.902046 20994 leader_election.cc:288] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:34.902182 20999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:34.903213 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:34.903172 20999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [term 1 LEADER]: Becoming Leader. State: Replica: 6f6f85f573ff431889577170d7814910, State: Running, Role: LEADER
I20260812 16:43:34.903702 20999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f85f573ff431889577170d7814910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46097 } }
I20260812 16:43:34.904219 20994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:34.909292 21000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f6f85f573ff431889577170d7814910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f85f573ff431889577170d7814910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46097 } } }
I20260812 16:43:34.910264 21000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:34.909646 21001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f6f85f573ff431889577170d7814910. Latest consensus state: current_term: 1 leader_uuid: "6f6f85f573ff431889577170d7814910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f85f573ff431889577170d7814910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46097 } } }
I20260812 16:43:34.910699 21001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:34.916476 21016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:34.922114 21016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:34.927047 21021 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 16:43:34.930563 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:34.910026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:34957 (local address 127.0.0.1:46097)
0812 16:43:34.910208 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:34.910214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:34.910495 (+ 281us) server_negotiation.cc:407] Connection header received
0812 16:43:34.910876 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:34.910884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:34.910974 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:34.911051 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:34.914606 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:34.915857 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:34.917536 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:34.918155 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:34.918290 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:34.921577 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:34.921584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:34.921589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:34.921635 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:34.926805 (+ 5170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:34.928250 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:34.928256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:34.928259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:34.928429 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:34.928926 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:34.928932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:34.928938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:34.929301 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:34.929605 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:34.929985 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 16:43:34.930182 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 16:43:34.933547 21016 catalog_manager.cc:1347] Generated new cluster ID: 0270d0afb0e649efb3ddfff3820afe60
I20260812 16:43:34.933655 21016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:34.945739 21016 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:34.947017 21016 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:34.957356 21016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f6f85f573ff431889577170d7814910: Generated new TSK 0
I20260812 16:43:34.958207 21016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:34.965368 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 20924
I20260812 16:43:34.966828 20989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:35.057667 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 20924
W20260812 16:43:35.072438 20909 connection.cc:535] client connection to 127.0.0.1:46097 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46097): Connection reset by peer (error 104)
2026-08-12T16:43:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:35 dist-test-slave-f7th krb5kdc[21026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:35 dist-test-slave-f7th krb5kdc[21026](info): set up 2 sockets
Aug 12 16:43:35 dist-test-slave-f7th krb5kdc[21026](info): commencing operation
krb5kdc: starting...
W20260812 16:43:36.112520 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:36 dist-test-slave-f7th krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:36.386694 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:36.730254 21039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:36.730592 21039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:36.730733 21039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:36.738183 21039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:36.738320 21039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:36.738402 21039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:36.738483 21039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:36.738554 21039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:36.748714 21039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:36.750074 21039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:36.751405 21039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:36.763530 21044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:36.763561 21045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:36.764253 21047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:36.764803 21039 server_base.cc:1034] running on GCE node
I20260812 16:43:36.765626 21039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:36.767114 21039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:36.768332 21039 hybrid_clock.cc:648] HybridClock initialized: now 1786553016768277 us; error 35 us; skew 500 ppm
I20260812 16:43:36.768697 21039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:36 dist-test-slave-f7th krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:36.771896 21039 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:36.773511 21039 webserver.cc:460] Webserver started at http://127.0.0.1:35615/ using document root <none> and password file <none>
I20260812 16:43:36.774232 21039 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:36.774427 21039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:36.774778 21039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:36.777953 21039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "7dc8170956cc46e4858558ebb8031387"
format_stamp: "Formatted at 2026-08-12 16:43:36 on dist-test-slave-f7th"
server_key: "d2a014a225086059eea0535b64f22da8"
server_key_iv: "52cf91799ee2f3a210490bb6cbe9e0c5"
server_key_version: "encryptionkey@0"
I20260812 16:43:36.778745 21039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "7dc8170956cc46e4858558ebb8031387"
format_stamp: "Formatted at 2026-08-12 16:43:36 on dist-test-slave-f7th"
server_key: "d2a014a225086059eea0535b64f22da8"
server_key_iv: "52cf91799ee2f3a210490bb6cbe9e0c5"
server_key_version: "encryptionkey@0"
I20260812 16:43:36.784232 21039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:43:36.787810 21055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:36.789299 21039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:43:36.789479 21039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "7dc8170956cc46e4858558ebb8031387"
format_stamp: "Formatted at 2026-08-12 16:43:36 on dist-test-slave-f7th"
server_key: "d2a014a225086059eea0535b64f22da8"
server_key_iv: "52cf91799ee2f3a210490bb6cbe9e0c5"
server_key_version: "encryptionkey@0"
I20260812 16:43:36.789669 21039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:36.832609 21039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:36.837185 21039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:36.837464 21039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:36.856007 21039 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35325
I20260812 16:43:36.856040 21108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35325 every 8 connection(s)
I20260812 16:43:36.857399 21039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:36.861400 21109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:36.865284 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 21039
I20260812 16:43:36.865487 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:36.866029 31990 external_mini_cluster.cc:1428] Setting key f88a3e880f224a73c48a79714ed80782
I20260812 16:43:36.874145 21109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387: Bootstrap starting.
I20260812 16:43:36.877470 21109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:36.878645 21109 log.cc:824] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:36.879066 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:36.868078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:43367 (local address 127.0.0.1:35325)
0812 16:43:36.868904 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:36.868917 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:36.868960 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:43:36.870694 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:36.870718 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:36.871159 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:36.871539 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:36.872127 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:36.873409 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:36.874743 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:36.875421 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:36.875649 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:36.876809 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:36.876817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:36.876830 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:36.877354 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:36.877437 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:36.877464 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:36.877606 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:36.877922 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 16:43:36.878073 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":348,"threads_started":1}
I20260812 16:43:36.881903 21109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387: No bootstrap required, opened a new log
I20260812 16:43:36.883337 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:36.892062 21109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc8170956cc46e4858558ebb8031387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260812 16:43:36.892468 21109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:36.892572 21109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc8170956cc46e4858558ebb8031387, State: Initialized, Role: FOLLOWER
I20260812 16:43:36.893183 21109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc8170956cc46e4858558ebb8031387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260812 16:43:36.893453 21109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:36.893592 21109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:36.893766 21109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:36.895679 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:36.888728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:59623 (local address 127.0.0.1:35325)
0812 16:43:36.888925 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:36.888931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:36.889427 (+ 496us) server_negotiation.cc:407] Connection header received
0812 16:43:36.889614 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:36.889624 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:36.889710 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:36.889839 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:36.890464 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:36.891568 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:36.893057 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:36.893560 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:36.893682 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:36.894850 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:36.894855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:36.894859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:36.894879 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:36.894943 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:36.894960 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:36.895064 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:36.895264 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 16:43:36.895324 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:43:36.896503 21109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc8170956cc46e4858558ebb8031387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260812 16:43:36.897105 21109 leader_election.cc:302] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dc8170956cc46e4858558ebb8031387; no voters:
I20260812 16:43:36.897689 21109 leader_election.cc:288] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:36.897814 21120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:36.898475 21120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [term 1 LEADER]: Becoming Leader. State: Replica: 7dc8170956cc46e4858558ebb8031387, State: Running, Role: LEADER
I20260812 16:43:36.899024 21120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc8170956cc46e4858558ebb8031387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260812 16:43:36.899852 21109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:36.901630 21121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dc8170956cc46e4858558ebb8031387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc8170956cc46e4858558ebb8031387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } } }
I20260812 16:43:36.902040 21121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:36.904307 21122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dc8170956cc46e4858558ebb8031387. Latest consensus state: current_term: 1 leader_uuid: "7dc8170956cc46e4858558ebb8031387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc8170956cc46e4858558ebb8031387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } } }
I20260812 16:43:36.904659 21122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:36.905539 21126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:36.910918 21126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:36.920181 21126 catalog_manager.cc:1347] Generated new cluster ID: 56f56ef044be4919a5e1e62042055190
I20260812 16:43:36.920357 21126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:36.943388 21126 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:36.945003 21126 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:36.959065 21126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dc8170956cc46e4858558ebb8031387: Generated new TSK 0
I20260812 16:43:36.960160 21126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:36.968626 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21039
W20260812 16:43:36.969866 21104 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.115:43367
W20260812 16:43:36.970692 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21039: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:36.970847 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 21039
W20260812 16:43:36.986568 21024 connection.cc:535] client connection to 127.0.0.1:35325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35325): Connection reset by peer (error 104)
2026-08-12T16:43:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:37 dist-test-slave-f7th krb5kdc[21142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:37 dist-test-slave-f7th krb5kdc[21142](info): set up 2 sockets
Aug 12 16:43:37 dist-test-slave-f7th krb5kdc[21142](info): commencing operation
krb5kdc: starting...
W20260812 16:43:38.019668 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:38 dist-test-slave-f7th krb5kdc[21142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:38.299441 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:38.635087 21156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:38.635377 21156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:38.635466 21156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:38.642810 21156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:38.642927 21156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:38.642971 21156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:38.643031 21156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:38.643074 21156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:38.653035 21156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:38.654281 21156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:38.655498 21156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:38.666534 21161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:38.666606 21164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:38.666582 21162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:38.668056 21156 server_base.cc:1034] running on GCE node
I20260812 16:43:38.668764 21156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:38.670064 21156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:38.671267 21156 hybrid_clock.cc:648] HybridClock initialized: now 1786553018671227 us; error 23 us; skew 500 ppm
I20260812 16:43:38.671630 21156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:38 dist-test-slave-f7th krb5kdc[21142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:38.678937 21156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:38.680454 21156 webserver.cc:460] Webserver started at http://127.0.0.1:43333/ using document root <none> and password file <none>
I20260812 16:43:38.681130 21156 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:38.681244 21156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:38.681522 21156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:38.684474 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "26c127f3fa1a401c86cf078ee969841d"
format_stamp: "Formatted at 2026-08-12 16:43:38 on dist-test-slave-f7th"
server_key: "b9fb5844b55ed2118f9f5bd523c61713"
server_key_iv: "d970d3ae1568eccaa15771259c9a08da"
server_key_version: "encryptionkey@0"
I20260812 16:43:38.685213 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "26c127f3fa1a401c86cf078ee969841d"
format_stamp: "Formatted at 2026-08-12 16:43:38 on dist-test-slave-f7th"
server_key: "b9fb5844b55ed2118f9f5bd523c61713"
server_key_iv: "d970d3ae1568eccaa15771259c9a08da"
server_key_version: "encryptionkey@0"
I20260812 16:43:38.690043 21156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 16:43:38.693509 21171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:38.694726 21156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:43:38.694882 21156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "26c127f3fa1a401c86cf078ee969841d"
format_stamp: "Formatted at 2026-08-12 16:43:38 on dist-test-slave-f7th"
server_key: "b9fb5844b55ed2118f9f5bd523c61713"
server_key_iv: "d970d3ae1568eccaa15771259c9a08da"
server_key_version: "encryptionkey@0"
I20260812 16:43:38.695052 21156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:38.700816 21156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:38.704818 21156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:38.705084 21156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:38.722389 21156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35903
I20260812 16:43:38.722572 21225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35903 every 8 connection(s)
I20260812 16:43:38.723863 21156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:38.727550 21226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:38.727721 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 21156
I20260812 16:43:38.727897 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:38.728360 31990 external_mini_cluster.cc:1428] Setting key 93d1726e9f74f83ba5b571ff09ec3d39
I20260812 16:43:38.736128 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:38.730151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:33629 (local address 127.0.0.1:35903)
0812 16:43:38.731048 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:38.731060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:38.731103 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:43:38.732773 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:38.732797 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:38.733146 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:38.733523 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:38.734527 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:38.734534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:38.734546 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:38.734563 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:38.734625 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:38.734647 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:38.734870 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:38.734995 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 16:43:38.735185 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":430,"threads_started":1}
I20260812 16:43:38.739873 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:38.741163 21226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d: Bootstrap starting.
I20260812 16:43:38.744385 21226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:38.745720 21226 log.cc:824] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:38.746626 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:38.743967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:34083 (local address 127.0.0.1:35903)
0812 16:43:38.744146 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:38.744152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:38.744672 (+ 520us) server_negotiation.cc:407] Connection header received
0812 16:43:38.744859 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:38.744862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:38.744914 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:38.744991 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:38.745936 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:38.745940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:38.745943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:38.745957 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:38.745995 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:38.746009 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:38.746094 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:38.746244 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 16:43:38.746308 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 16:43:38.748612 21226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d: No bootstrap required, opened a new log
I20260812 16:43:38.755672 21226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c127f3fa1a401c86cf078ee969841d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35903 } }
I20260812 16:43:38.756071 21226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:38.756146 21226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c127f3fa1a401c86cf078ee969841d, State: Initialized, Role: FOLLOWER
I20260812 16:43:38.757092 21226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c127f3fa1a401c86cf078ee969841d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35903 } }
I20260812 16:43:38.757453 21226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:38.757615 21226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:38.757797 21226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:38.760159 21226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c127f3fa1a401c86cf078ee969841d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35903 } }
I20260812 16:43:38.760721 21226 leader_election.cc:302] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26c127f3fa1a401c86cf078ee969841d; no voters:
I20260812 16:43:38.761345 21226 leader_election.cc:288] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:38.761593 21236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:38.762302 21236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [term 1 LEADER]: Becoming Leader. State: Replica: 26c127f3fa1a401c86cf078ee969841d, State: Running, Role: LEADER
I20260812 16:43:38.762917 21236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c127f3fa1a401c86cf078ee969841d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35903 } }
I20260812 16:43:38.763521 21226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:38.765648 21237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26c127f3fa1a401c86cf078ee969841d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c127f3fa1a401c86cf078ee969841d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35903 } } }
I20260812 16:43:38.766193 21237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:38.766403 21238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c127f3fa1a401c86cf078ee969841d. Latest consensus state: current_term: 1 leader_uuid: "26c127f3fa1a401c86cf078ee969841d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c127f3fa1a401c86cf078ee969841d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35903 } } }
I20260812 16:43:38.766963 21238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:38.769170 21244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:38.773878 21244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:38.782908 21244 catalog_manager.cc:1347] Generated new cluster ID: 9c9cef088e984c13bee1bc5ffc45aefc
I20260812 16:43:38.783043 21244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:38.795456 21244 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:38.796017 21244 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:38.808157 21244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26c127f3fa1a401c86cf078ee969841d: Generated new TSK 0
I20260812 16:43:38.809186 21244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:38.817997 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21156
W20260812 16:43:38.819562 21221 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.115:33629
W20260812 16:43:38.820324 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21156: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:38.820447 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 21156
2026-08-12T16:43:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1848 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:38 dist-test-slave-f7th krb5kdc[21257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:38 dist-test-slave-f7th krb5kdc[21257](info): set up 2 sockets
Aug 12 16:43:38 dist-test-slave-f7th krb5kdc[21257](info): commencing operation
krb5kdc: starting...
W20260812 16:43:39.847767 31990 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:40 dist-test-slave-f7th krb5kdc[21257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:40.125572 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:40.470644 21271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:40.470935 21271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:40.471030 21271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:40.478228 21271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:40.478345 21271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:40.478401 21271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:40.478468 21271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:40.478513 21271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:40.488277 21271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:40.489535 21271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:40.490764 21271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:40.501745 21276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:40.501719 21277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:40.501780 21279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:40.502566 21271 server_base.cc:1034] running on GCE node
I20260812 16:43:40.503360 21271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:40.504810 21271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:40.506027 21271 hybrid_clock.cc:648] HybridClock initialized: now 1786553020505962 us; error 40 us; skew 500 ppm
I20260812 16:43:40.506350 21271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:40 dist-test-slave-f7th krb5kdc[21257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:40.509388 21271 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:40.510823 21271 webserver.cc:460] Webserver started at http://127.0.0.1:43627/ using document root <none> and password file <none>
I20260812 16:43:40.511482 21271 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:40.511631 21271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:40.511916 21271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:40.514902 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "a2d0afd4fdc944a9bc8f26218e043be7"
format_stamp: "Formatted at 2026-08-12 16:43:40 on dist-test-slave-f7th"
server_key: "7b81f5651e72a9cc293f7dc75af75761"
server_key_iv: "14ff0ee7f965721f4d7ec66078122633"
server_key_version: "encryptionkey@0"
I20260812 16:43:40.515712 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "a2d0afd4fdc944a9bc8f26218e043be7"
format_stamp: "Formatted at 2026-08-12 16:43:40 on dist-test-slave-f7th"
server_key: "7b81f5651e72a9cc293f7dc75af75761"
server_key_iv: "14ff0ee7f965721f4d7ec66078122633"
server_key_version: "encryptionkey@0"
I20260812 16:43:40.520845 21271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 16:43:40.524608 21286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:40.526125 21271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:43:40.526316 21271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "a2d0afd4fdc944a9bc8f26218e043be7"
format_stamp: "Formatted at 2026-08-12 16:43:40 on dist-test-slave-f7th"
server_key: "7b81f5651e72a9cc293f7dc75af75761"
server_key_iv: "14ff0ee7f965721f4d7ec66078122633"
server_key_version: "encryptionkey@0"
I20260812 16:43:40.526520 21271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:40.555027 21271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:40.559921 21271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:40.560218 21271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:40.579954 21271 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38811
I20260812 16:43:40.579972 21337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38811 every 8 connection(s)
I20260812 16:43:40.581406 21271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:40.584846 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 21271
I20260812 16:43:40.585037 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:40.585542 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:40.585616 31990 external_mini_cluster.cc:1428] Setting key 51abdf4f345883e6031557ed70dd7d4b
I20260812 16:43:40.599893 21338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7: Bootstrap starting.
Aug 12 16:43:40 dist-test-slave-f7th krb5kdc[21257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 16:43:40.603379 21338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:40.604728 21338 log.cc:824] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:40.607661 21338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7: No bootstrap required, opened a new log
I20260812 16:43:40.608985 21341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:40.587360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:41051 (local address 127.0.0.1:38811)
0812 16:43:40.588170 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:40.588181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:40.588225 (+ 44us) server_negotiation.cc:407] Connection header received
0812 16:43:40.590213 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:40.590262 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:40.590683 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:40.591034 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:40.592760 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:40.594074 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:40.595874 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:40.596667 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:40.596811 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:40.601589 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:40.601597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:40.601611 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:40.601660 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:40.604779 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:40.605625 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:40.605632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:40.605639 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:40.605818 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:40.606294 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:40.606298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:40.606300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:40.606825 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:40.607150 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:40.607691 (+ 541us) server_negotiation.cc:299] Negotiation successful
0812 16:43:40.608007 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":348,"threads_started":1}
I20260812 16:43:40.612980 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:40.616117 21338 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d0afd4fdc944a9bc8f26218e043be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38811 } }
I20260812 16:43:40.616526 21338 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:40.616633 21338 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d0afd4fdc944a9bc8f26218e043be7, State: Initialized, Role: FOLLOWER
I20260812 16:43:40.617316 21338 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d0afd4fdc944a9bc8f26218e043be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38811 } }
I20260812 16:43:40.617635 21338 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:40.617794 21338 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:40.617959 21338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:40.620672 21338 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d0afd4fdc944a9bc8f26218e043be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38811 } }
I20260812 16:43:40.621263 21338 leader_election.cc:302] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d0afd4fdc944a9bc8f26218e043be7; no voters:
I20260812 16:43:40.621869 21338 leader_election.cc:288] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:40.622013 21349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:40.622625 21349 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [term 1 LEADER]: Becoming Leader. State: Replica: a2d0afd4fdc944a9bc8f26218e043be7, State: Running, Role: LEADER
I20260812 16:43:40.623220 21349 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d0afd4fdc944a9bc8f26218e043be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38811 } }
I20260812 16:43:40.625492 21338 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:40.626111 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2d0afd4fdc944a9bc8f26218e043be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d0afd4fdc944a9bc8f26218e043be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38811 } } }
I20260812 16:43:40.626500 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:40.630411 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d0afd4fdc944a9bc8f26218e043be7. Latest consensus state: current_term: 1 leader_uuid: "a2d0afd4fdc944a9bc8f26218e043be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d0afd4fdc944a9bc8f26218e043be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38811 } } }
I20260812 16:43:40.630712 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:40.632359 21353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:40.634689 21341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:40.617277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:51011 (local address 127.0.0.1:38811)
0812 16:43:40.617470 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:40.617475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:40.618100 (+ 625us) server_negotiation.cc:407] Connection header received
0812 16:43:40.618256 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:40.618261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:40.618342 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:40.618449 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:40.619839 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:40.620895 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:40.622499 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:40.623181 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:40.623288 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:40.626846 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:40.626853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:40.626858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:40.626899 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:40.630947 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:40.632068 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:40.632074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:40.632079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:40.632256 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:40.632826 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:40.632832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:40.632836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:40.633347 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:40.633675 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:40.634055 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 16:43:40.634272 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 16:43:40.638020 21353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:40.648550 21353 catalog_manager.cc:1347] Generated new cluster ID: 4c6b495dee7a48c8a06ead46329cee2e
I20260812 16:43:40.648711 21353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:40.663884 21353 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:40.665401 21353 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:40.688076 21353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d0afd4fdc944a9bc8f26218e043be7: Generated new TSK 0
I20260812 16:43:40.689190 21353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:40.696606 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21271
I20260812 16:43:40.698619 21333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:40.796514 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 21271
W20260812 16:43:40.811386 21255 connection.cc:535] client connection to 127.0.0.1:38811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38811): Connection reset by peer (error 104)
2026-08-12T16:43:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:40 dist-test-slave-f7th krb5kdc[21371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:40 dist-test-slave-f7th krb5kdc[21371](info): set up 2 sockets
Aug 12 16:43:40 dist-test-slave-f7th krb5kdc[21371](info): commencing operation
krb5kdc: starting...
W20260812 16:43:41.843724 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.010s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:42 dist-test-slave-f7th krb5kdc[21371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:42.118814 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:42.460603 21384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:42.460923 21384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:42.461061 21384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:42.468559 21384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:42.468703 21384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:42.468786 21384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:42.468866 21384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:42.468925 21384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:42.479768 21384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:42.481178 21384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:42.482525 21384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:42.494616 21389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:42.494659 21392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:42.494638 21390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:42.495383 21384 server_base.cc:1034] running on GCE node
I20260812 16:43:42.496235 21384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:42.497685 21384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:42.498895 21384 hybrid_clock.cc:648] HybridClock initialized: now 1786553022498853 us; error 33 us; skew 500 ppm
I20260812 16:43:42.499267 21384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:42 dist-test-slave-f7th krb5kdc[21371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:42.502457 21384 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:42.504010 21384 webserver.cc:460] Webserver started at http://127.0.0.1:39031/ using document root <none> and password file <none>
I20260812 16:43:42.504712 21384 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:42.504832 21384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:42.505132 21384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:42.508175 21384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "7dcd823ab8d44b2e8784ebe830e47028"
format_stamp: "Formatted at 2026-08-12 16:43:42 on dist-test-slave-f7th"
server_key: "439e3e304f954cbb1550d23eebe7d1e3"
server_key_iv: "f11d278b506b3cd12fc27693f5a620ce"
server_key_version: "encryptionkey@0"
I20260812 16:43:42.508939 21384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "7dcd823ab8d44b2e8784ebe830e47028"
format_stamp: "Formatted at 2026-08-12 16:43:42 on dist-test-slave-f7th"
server_key: "439e3e304f954cbb1550d23eebe7d1e3"
server_key_iv: "f11d278b506b3cd12fc27693f5a620ce"
server_key_version: "encryptionkey@0"
I20260812 16:43:42.514281 21384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:43:42.517980 21399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:42.519491 21384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:43:42.519688 21384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "7dcd823ab8d44b2e8784ebe830e47028"
format_stamp: "Formatted at 2026-08-12 16:43:42 on dist-test-slave-f7th"
server_key: "439e3e304f954cbb1550d23eebe7d1e3"
server_key_iv: "f11d278b506b3cd12fc27693f5a620ce"
server_key_version: "encryptionkey@0"
I20260812 16:43:42.519887 21384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:42.536917 21384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:42.541522 21384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:42.541816 21384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:42.560029 21384 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36093
I20260812 16:43:42.560073 21453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36093 every 8 connection(s)
I20260812 16:43:42.561499 21384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:42.565249 21454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:42.567298 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 21384
I20260812 16:43:42.567451 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:42.567915 31990 external_mini_cluster.cc:1428] Setting key 69b4141a65bf66913f7af814c1cdfbc9
I20260812 16:43:42.577519 21454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028: Bootstrap starting.
I20260812 16:43:42.581198 21454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:42.582052 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:42.569870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:46913 (local address 127.0.0.1:36093)
0812 16:43:42.570752 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:42.570764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:42.570807 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:43:42.572605 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:42.572630 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:42.573051 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:42.573373 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:42.574027 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:42.575458 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:42.577342 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:42.578080 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:42.578208 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:42.579631 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:42.579643 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:42.579662 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:42.580250 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:42.580335 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:42.580360 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:42.580493 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:42.580776 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 16:43:42.580963 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":376,"threads_started":1}
I20260812 16:43:42.582340 21454 log.cc:824] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:42.585659 21454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028: No bootstrap required, opened a new log
I20260812 16:43:42.586350 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:42.593827 21454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd823ab8d44b2e8784ebe830e47028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260812 16:43:42.594221 21454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:42.594352 21454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcd823ab8d44b2e8784ebe830e47028, State: Initialized, Role: FOLLOWER
I20260812 16:43:42.595067 21454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd823ab8d44b2e8784ebe830e47028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260812 16:43:42.595371 21454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:42.595526 21454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:42.595710 21454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:42.597126 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:42.590597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:41929 (local address 127.0.0.1:36093)
0812 16:43:42.590747 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:42.590751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:42.591285 (+ 534us) server_negotiation.cc:407] Connection header received
0812 16:43:42.591498 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:42.591502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:42.591578 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:42.591695 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:42.592205 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:42.593007 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:42.594295 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:42.594944 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:42.595105 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:42.596242 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:42.596247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:42.596250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:42.596269 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:42.596311 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:42.596330 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:42.596415 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:42.596724 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 16:43:42.596793 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 16:43:42.598812 21454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd823ab8d44b2e8784ebe830e47028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260812 16:43:42.599334 21454 leader_election.cc:302] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dcd823ab8d44b2e8784ebe830e47028; no voters:
I20260812 16:43:42.599958 21454 leader_election.cc:288] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:42.600055 21464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:42.602079 21454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:42.603997 21464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [term 1 LEADER]: Becoming Leader. State: Replica: 7dcd823ab8d44b2e8784ebe830e47028, State: Running, Role: LEADER
I20260812 16:43:42.604620 21464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd823ab8d44b2e8784ebe830e47028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260812 16:43:42.607445 21466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dcd823ab8d44b2e8784ebe830e47028. Latest consensus state: current_term: 1 leader_uuid: "7dcd823ab8d44b2e8784ebe830e47028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd823ab8d44b2e8784ebe830e47028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } } }
I20260812 16:43:42.608021 21466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:42.608392 21465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dcd823ab8d44b2e8784ebe830e47028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd823ab8d44b2e8784ebe830e47028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } } }
I20260812 16:43:42.608640 21465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:42.610211 21477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:42.616130 21477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:42.626220 21477 catalog_manager.cc:1347] Generated new cluster ID: fe65b8260abe44e49eca8872bf66000b
I20260812 16:43:42.626341 21477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:42.640034 21477 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:42.641443 21477 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:42.655196 21477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dcd823ab8d44b2e8784ebe830e47028: Generated new TSK 0
I20260812 16:43:42.656332 21477 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:42.672022 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21384
W20260812 16:43:42.673177 21448 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.115:46913
W20260812 16:43:42.673982 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21384: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:42.674122 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 21384
W20260812 16:43:42.689910 21369 connection.cc:535] client connection to 127.0.0.1:36093 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36093): Connection reset by peer (error 104)
2026-08-12T16:43:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:42 dist-test-slave-f7th krb5kdc[21487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:42 dist-test-slave-f7th krb5kdc[21487](info): set up 2 sockets
Aug 12 16:43:42 dist-test-slave-f7th krb5kdc[21487](info): commencing operation
krb5kdc: starting...
W20260812 16:43:43.723502 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:43 dist-test-slave-f7th krb5kdc[21487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 16:43:43.999589 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:44.339802 21500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:44.340138 21500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:44.340279 21500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:44.347618 21500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:44.347750 21500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:44.347832 21500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:44.347915 21500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:44.347973 21500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:44.357908 21500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:44.359236 21500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:44.360549 21500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:44.375571 21505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:44.375461 21506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:44.375496 21508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:44.376698 21500 server_base.cc:1034] running on GCE node
I20260812 16:43:44.377449 21500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:44.378922 21500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:44.380132 21500 hybrid_clock.cc:648] HybridClock initialized: now 1786553024380078 us; error 50 us; skew 500 ppm
I20260812 16:43:44.380470 21500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:44 dist-test-slave-f7th krb5kdc[21487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:44.388880 21500 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 16:43:44.390574 21500 webserver.cc:460] Webserver started at http://127.0.0.1:43475/ using document root <none> and password file <none>
I20260812 16:43:44.391302 21500 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:44.391441 21500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:44.391803 21500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:44.394970 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "d9a935f26d4e43eca98cbdd7eb0a6303"
format_stamp: "Formatted at 2026-08-12 16:43:44 on dist-test-slave-f7th"
server_key: "0e42b9a77de70f8d2e19efbf70910382"
server_key_iv: "58f84b644334539bb85ac350bff041ed"
server_key_version: "encryptionkey@0"
I20260812 16:43:44.395833 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "d9a935f26d4e43eca98cbdd7eb0a6303"
format_stamp: "Formatted at 2026-08-12 16:43:44 on dist-test-slave-f7th"
server_key: "0e42b9a77de70f8d2e19efbf70910382"
server_key_iv: "58f84b644334539bb85ac350bff041ed"
server_key_version: "encryptionkey@0"
I20260812 16:43:44.401527 21500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 16:43:44.405170 21515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:44.406566 21500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 16:43:44.406735 21500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "d9a935f26d4e43eca98cbdd7eb0a6303"
format_stamp: "Formatted at 2026-08-12 16:43:44 on dist-test-slave-f7th"
server_key: "0e42b9a77de70f8d2e19efbf70910382"
server_key_iv: "58f84b644334539bb85ac350bff041ed"
server_key_version: "encryptionkey@0"
I20260812 16:43:44.406930 21500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:44.412623 21500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:44.416815 21500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:44.417092 21500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:44.435182 21500 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33357
I20260812 16:43:44.435232 21566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33357 every 8 connection(s)
I20260812 16:43:44.436611 21500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:44.437721 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 21500
I20260812 16:43:44.437909 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:44.438503 31990 external_mini_cluster.cc:1428] Setting key 2468938d57cd25a70433c5955abb29a8
I20260812 16:43:44.441396 21567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:44.446318 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:44.440451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:33931 (local address 127.0.0.1:33357)
0812 16:43:44.441223 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:44.441234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:44.441274 (+ 40us) server_negotiation.cc:407] Connection header received
0812 16:43:44.442895 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:44.442917 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:44.443223 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:44.443564 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:44.444546 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:44.444554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:44.444566 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:44.444580 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:44.444660 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:44.444685 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:44.444914 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:44.445062 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 16:43:44.445262 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":304,"threads_started":1}
I20260812 16:43:44.450078 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:44.454334 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303: Bootstrap starting.
I20260812 16:43:44.457580 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:44.455294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:32787 (local address 127.0.0.1:33357)
0812 16:43:44.455467 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:44.455473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:44.455794 (+ 321us) server_negotiation.cc:407] Connection header received
0812 16:43:44.455931 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:44.455936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:44.456008 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:44.456129 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:44.456983 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:44.456988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:44.456993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:44.457011 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:44.457057 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:44.457078 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:44.457203 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:44.457248 (+ 45us) server_negotiation.cc:299] Negotiation successful
0812 16:43:44.457307 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 16:43:44.458089 21567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:44.459280 21567 log.cc:824] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:44.461970 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303: No bootstrap required, opened a new log
I20260812 16:43:44.469317 21567 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a935f26d4e43eca98cbdd7eb0a6303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33357 } }
I20260812 16:43:44.469681 21567 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:44.469753 21567 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a935f26d4e43eca98cbdd7eb0a6303, State: Initialized, Role: FOLLOWER
I20260812 16:43:44.470256 21567 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a935f26d4e43eca98cbdd7eb0a6303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33357 } }
I20260812 16:43:44.470448 21567 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:44.470547 21567 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:44.470670 21567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:44.472808 21567 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a935f26d4e43eca98cbdd7eb0a6303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33357 } }
I20260812 16:43:44.473259 21567 leader_election.cc:302] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9a935f26d4e43eca98cbdd7eb0a6303; no voters:
I20260812 16:43:44.473809 21567 leader_election.cc:288] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:44.473965 21578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:44.474597 21578 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [term 1 LEADER]: Becoming Leader. State: Replica: d9a935f26d4e43eca98cbdd7eb0a6303, State: Running, Role: LEADER
I20260812 16:43:44.475230 21578 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a935f26d4e43eca98cbdd7eb0a6303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33357 } }
I20260812 16:43:44.476040 21567 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:44.477962 21579 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9a935f26d4e43eca98cbdd7eb0a6303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a935f26d4e43eca98cbdd7eb0a6303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33357 } } }
I20260812 16:43:44.478313 21580 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a935f26d4e43eca98cbdd7eb0a6303. Latest consensus state: current_term: 1 leader_uuid: "d9a935f26d4e43eca98cbdd7eb0a6303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a935f26d4e43eca98cbdd7eb0a6303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33357 } } }
I20260812 16:43:44.478608 21579 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:44.478609 21580 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:44.481045 21586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:44.486142 21586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:44.494884 21586 catalog_manager.cc:1347] Generated new cluster ID: 44c7ebf987224d2e975e1f0cb768f02e
I20260812 16:43:44.494992 21586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:44.504545 21586 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:44.504949 21586 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:44.512639 21586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9a935f26d4e43eca98cbdd7eb0a6303: Generated new TSK 0
I20260812 16:43:44.513577 21586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:44.522933 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21500
W20260812 16:43:44.523959 21562 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.115:33931
W20260812 16:43:44.524632 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21500: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:44.524757 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 21500
2026-08-12T16:43:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:44 dist-test-slave-f7th krb5kdc[21599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:44 dist-test-slave-f7th krb5kdc[21599](info): set up 2 sockets
Aug 12 16:43:44 dist-test-slave-f7th krb5kdc[21599](info): commencing operation
krb5kdc: starting...
W20260812 16:43:45.571718 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:45 dist-test-slave-f7th krb5kdc[21599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:45.845156 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:41219
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:41219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:46.171859 21612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:46.172153 21612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:46.172272 21612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:46.180120 21612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:46.180243 21612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:46.180305 21612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:46.180382 21612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:46.180442 21612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:46.190793 21612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:41219
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:46.192265 21612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:46.193639 21612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:46.206153 21617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:46.206178 21620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:46.206108 21618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:46.206784 21612 server_base.cc:1034] running on GCE node
I20260812 16:43:46.207690 21612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:46.209189 21612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:46.210439 21612 hybrid_clock.cc:648] HybridClock initialized: now 1786553026210358 us; error 48 us; skew 500 ppm
I20260812 16:43:46.210811 21612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:46 dist-test-slave-f7th krb5kdc[21599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553026, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:46.214032 21612 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:46.215715 21612 webserver.cc:460] Webserver started at http://127.31.61.190:41207/ using document root <none> and password file <none>
I20260812 16:43:46.216452 21612 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:46.216580 21612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:46.216886 21612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:46.220080 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "55002b005f3843adac10ed0c56dcb87c"
format_stamp: "Formatted at 2026-08-12 16:43:46 on dist-test-slave-f7th"
server_key: "a5baa0dfb772182f392532fa5bbad47e"
server_key_iv: "bc17f6faaff72606661741782be29bd8"
server_key_version: "encryptionkey@0"
I20260812 16:43:46.220889 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "55002b005f3843adac10ed0c56dcb87c"
format_stamp: "Formatted at 2026-08-12 16:43:46 on dist-test-slave-f7th"
server_key: "a5baa0dfb772182f392532fa5bbad47e"
server_key_iv: "bc17f6faaff72606661741782be29bd8"
server_key_version: "encryptionkey@0"
I20260812 16:43:46.227411 21612 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 16:43:46.231624 21627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:46.233204 21612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 16:43:46.233417 21612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "55002b005f3843adac10ed0c56dcb87c"
format_stamp: "Formatted at 2026-08-12 16:43:46 on dist-test-slave-f7th"
server_key: "a5baa0dfb772182f392532fa5bbad47e"
server_key_iv: "bc17f6faaff72606661741782be29bd8"
server_key_version: "encryptionkey@0"
I20260812 16:43:46.233634 21612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:46.249516 21612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:46.253686 21612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:46.253961 21612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:46.272821 21612 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:41219
I20260812 16:43:46.272855 21679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:41219 every 8 connection(s)
I20260812 16:43:46.274262 21612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:46.278688 21680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:46.281229 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 21612
I20260812 16:43:46.281412 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:46.281960 31990 external_mini_cluster.cc:1428] Setting key 8f908af59d583205130f18d07190fe54
I20260812 16:43:46.294217 21680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c: Bootstrap starting.
Aug 12 16:43:46 dist-test-slave-f7th krb5kdc[21599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:46.298575 21680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:46.299839 21680 log.cc:824] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:46.303319 21680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c: No bootstrap required, opened a new log
I20260812 16:43:46.308487 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:46.283932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47505 (local address 127.31.61.190:41219)
0812 16:43:46.284818 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:46.284835 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:46.284890 (+ 55us) server_negotiation.cc:407] Connection header received
0812 16:43:46.286537 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:46.286571 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:46.287016 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:46.287353 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:46.289151 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:46.290545 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:46.292371 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:46.293080 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:46.293214 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:46.298441 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:46.298451 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:46.298475 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:46.298527 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:46.301964 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:46.303058 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:46.303069 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:46.303078 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:46.303309 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:46.304077 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:46.304086 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:46.304089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:46.305214 (+ 1125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:46.305754 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:46.306286 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 16:43:46.306868 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":381,"threads_started":1}
I20260812 16:43:46.312789 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:46.313060 21680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55002b005f3843adac10ed0c56dcb87c" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41219 } }
I20260812 16:43:46.313491 21680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:46.313598 21680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55002b005f3843adac10ed0c56dcb87c, State: Initialized, Role: FOLLOWER
I20260812 16:43:46.314224 21680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55002b005f3843adac10ed0c56dcb87c" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41219 } }
I20260812 16:43:46.314488 21680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:46.314625 21680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:46.314769 21680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:46.317952 21680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55002b005f3843adac10ed0c56dcb87c" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41219 } }
I20260812 16:43:46.318477 21680 leader_election.cc:302] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55002b005f3843adac10ed0c56dcb87c; no voters:
I20260812 16:43:46.319097 21680 leader_election.cc:288] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:46.319211 21690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:46.320267 21690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [term 1 LEADER]: Becoming Leader. State: Replica: 55002b005f3843adac10ed0c56dcb87c, State: Running, Role: LEADER
I20260812 16:43:46.320864 21690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55002b005f3843adac10ed0c56dcb87c" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41219 } }
I20260812 16:43:46.321599 21680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:46.323755 21691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55002b005f3843adac10ed0c56dcb87c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55002b005f3843adac10ed0c56dcb87c" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41219 } } }
I20260812 16:43:46.324158 21691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:46.328676 21692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55002b005f3843adac10ed0c56dcb87c. Latest consensus state: current_term: 1 leader_uuid: "55002b005f3843adac10ed0c56dcb87c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55002b005f3843adac10ed0c56dcb87c" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41219 } } }
I20260812 16:43:46.329015 21692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:46.332968 21695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:46.335901 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:46.317935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51119 (local address 127.31.61.190:41219)
0812 16:43:46.318139 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:46.318145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:46.318586 (+ 441us) server_negotiation.cc:407] Connection header received
0812 16:43:46.318763 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:46.318767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:46.318828 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:46.318919 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:46.320763 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:46.321708 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:46.323147 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:46.323835 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:46.323944 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:46.327710 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:46.327718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:46.327724 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:46.327772 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:46.331962 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:46.332994 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:46.333010 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:46.333015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:46.333199 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:46.333770 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:46.333775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:46.333778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:46.334361 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:46.334680 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:46.335201 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 16:43:46.335456 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 16:43:46.339448 21695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:46.349347 21695 catalog_manager.cc:1347] Generated new cluster ID: 3539d05bc4564881af609d13d1371bd2
I20260812 16:43:46.349534 21695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:46.362275 21695 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:46.363690 21695 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:46.374050 21695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55002b005f3843adac10ed0c56dcb87c: Generated new TSK 0
I20260812 16:43:46.375174 21695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:46.382758 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21612
I20260812 16:43:46.384209 21674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:46.475699 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 21612
W20260812 16:43:46.490656 21597 connection.cc:535] client connection to 127.31.61.190:41219 recv error: Network error: failed to read from TLS socket (remote: 127.31.61.190:41219): Connection reset by peer (error 104)
2026-08-12T16:43:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:46 dist-test-slave-f7th krb5kdc[21713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:46 dist-test-slave-f7th krb5kdc[21713](info): set up 2 sockets
Aug 12 16:43:46 dist-test-slave-f7th krb5kdc[21713](info): commencing operation
krb5kdc: starting...
W20260812 16:43:47.519966 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:47 dist-test-slave-f7th krb5kdc[21713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:47.795560 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:41329
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:41329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:48.133237 21727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:48.133564 21727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:48.133665 21727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:48.141470 21727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:48.141613 21727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:48.141665 21727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:48.141752 21727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:48.141816 21727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:48.152079 21727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:41329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:41329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:48.153967 21727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:48.155649 21727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:48.167690 21732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:48.167699 21735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:48.167703 21733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:48.168358 21727 server_base.cc:1034] running on GCE node
I20260812 16:43:48.169188 21727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:48.170841 21727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:48.172091 21727 hybrid_clock.cc:648] HybridClock initialized: now 1786553028172033 us; error 53 us; skew 500 ppm
I20260812 16:43:48.172454 21727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:48 dist-test-slave-f7th krb5kdc[21713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:48.175747 21727 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:48.177345 21727 webserver.cc:460] Webserver started at http://127.31.61.190:42685/ using document root <none> and password file <none>
I20260812 16:43:48.178032 21727 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:48.178138 21727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:48.178396 21727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:48.181517 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "c9796d01ebc444b39a6aedeb88e6a2f8"
format_stamp: "Formatted at 2026-08-12 16:43:48 on dist-test-slave-f7th"
server_key: "cad8ea4bb5a3c2ac75dac2194e733d08"
server_key_iv: "dd5de9c59eb44f867f4c45b2a7f8dd7e"
server_key_version: "encryptionkey@0"
I20260812 16:43:48.182243 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "c9796d01ebc444b39a6aedeb88e6a2f8"
format_stamp: "Formatted at 2026-08-12 16:43:48 on dist-test-slave-f7th"
server_key: "cad8ea4bb5a3c2ac75dac2194e733d08"
server_key_iv: "dd5de9c59eb44f867f4c45b2a7f8dd7e"
server_key_version: "encryptionkey@0"
I20260812 16:43:48.187636 21727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 16:43:48.191246 21742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:48.192723 21727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:48.192905 21727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "c9796d01ebc444b39a6aedeb88e6a2f8"
format_stamp: "Formatted at 2026-08-12 16:43:48 on dist-test-slave-f7th"
server_key: "cad8ea4bb5a3c2ac75dac2194e733d08"
server_key_iv: "dd5de9c59eb44f867f4c45b2a7f8dd7e"
server_key_version: "encryptionkey@0"
I20260812 16:43:48.193113 21727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:48.206642 21727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:48.211108 21727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:48.211369 21727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:48.229447 21727 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:41329
I20260812 16:43:48.229455 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:41329 every 8 connection(s)
I20260812 16:43:48.230857 21727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:48.232151 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 21727
I20260812 16:43:48.232321 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:48.232888 31990 external_mini_cluster.cc:1428] Setting key e0f2c0619f89e8865ff0e83364591722
I20260812 16:43:48.235494 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:48.247105 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:48.234703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57883 (local address 127.31.61.190:41329)
0812 16:43:48.235510 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:48.235521 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:48.235568 (+ 47us) server_negotiation.cc:407] Connection header received
0812 16:43:48.237406 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:48.237429 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:48.237821 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:48.238159 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:48.238972 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:48.240484 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:48.242359 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:48.243093 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:48.243238 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:48.244611 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:48.244623 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:48.244637 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:48.245263 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:48.245377 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:48.245408 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:48.245569 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:48.245840 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 16:43:48.246043 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":366,"threads_started":1}
I20260812 16:43:48.247962 21798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8: Bootstrap starting.
I20260812 16:43:48.251060 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:48.251051 21798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:48.252270 21798 log.cc:824] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:48.255661 21798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8: No bootstrap required, opened a new log
I20260812 16:43:48.262892 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:48.256015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54533 (local address 127.31.61.190:41329)
0812 16:43:48.256230 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:48.256236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:48.256716 (+ 480us) server_negotiation.cc:407] Connection header received
0812 16:43:48.256844 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:48.256850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:48.256921 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:48.257033 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:48.257521 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:48.258694 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:48.260111 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:48.260856 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:48.260995 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:48.262103 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:48.262109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:48.262114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:48.262139 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:48.262205 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:48.262229 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:48.262343 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:48.262474 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 16:43:48.262538 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:43:48.265157 21798 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796d01ebc444b39a6aedeb88e6a2f8" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41329 } }
I20260812 16:43:48.265522 21798 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:48.265599 21798 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9796d01ebc444b39a6aedeb88e6a2f8, State: Initialized, Role: FOLLOWER
I20260812 16:43:48.266103 21798 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796d01ebc444b39a6aedeb88e6a2f8" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41329 } }
I20260812 16:43:48.266295 21798 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:48.266391 21798 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:48.266520 21798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:48.269012 21798 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796d01ebc444b39a6aedeb88e6a2f8" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41329 } }
I20260812 16:43:48.269470 21798 leader_election.cc:302] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9796d01ebc444b39a6aedeb88e6a2f8; no voters:
I20260812 16:43:48.270030 21798 leader_election.cc:288] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:48.270156 21808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:48.270787 21808 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [term 1 LEADER]: Becoming Leader. State: Replica: c9796d01ebc444b39a6aedeb88e6a2f8, State: Running, Role: LEADER
I20260812 16:43:48.271277 21808 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796d01ebc444b39a6aedeb88e6a2f8" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41329 } }
I20260812 16:43:48.272008 21798 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:48.273792 21810 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9796d01ebc444b39a6aedeb88e6a2f8. Latest consensus state: current_term: 1 leader_uuid: "c9796d01ebc444b39a6aedeb88e6a2f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796d01ebc444b39a6aedeb88e6a2f8" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41329 } } }
I20260812 16:43:48.274222 21810 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:48.276599 21809 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9796d01ebc444b39a6aedeb88e6a2f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9796d01ebc444b39a6aedeb88e6a2f8" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 41329 } } }
I20260812 16:43:48.276897 21809 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:48.277537 21816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:48.283177 21816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:48.291675 21816 catalog_manager.cc:1347] Generated new cluster ID: c38e1742f1984b4bb32860b6597ad2d1
I20260812 16:43:48.291790 21816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:48.315320 21816 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:48.316784 21816 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:48.327692 21816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9796d01ebc444b39a6aedeb88e6a2f8: Generated new TSK 0
I20260812 16:43:48.328509 21816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:48.343237 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21727
W20260812 16:43:48.344551 21793 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57883
W20260812 16:43:48.345418 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21727: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:48.345597 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 21727
W20260812 16:43:48.360698 21711 connection.cc:535] client connection to 127.31.61.190:41329 recv error: Network error: failed to read from TLS socket (remote: 127.31.61.190:41329): Connection reset by peer (error 104)
2026-08-12T16:43:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:48 dist-test-slave-f7th krb5kdc[21829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:48 dist-test-slave-f7th krb5kdc[21829](info): set up 2 sockets
Aug 12 16:43:48 dist-test-slave-f7th krb5kdc[21829](info): commencing operation
krb5kdc: starting...
W20260812 16:43:49.391567 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:49 dist-test-slave-f7th krb5kdc[21829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:49.667243 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36227
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:50.001542 21842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:50.001893 21842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:50.002041 21842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:50.009475 21842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:50.009629 21842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:50.009712 21842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:50.009801 21842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:50.009865 21842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:50.020063 21842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:50.021477 21842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:50.022996 21842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:50.034765 21848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:50.034827 21849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:50.034806 21851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:50.035701 21842 server_base.cc:1034] running on GCE node
I20260812 16:43:50.036517 21842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:50.037926 21842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:50.039132 21842 hybrid_clock.cc:648] HybridClock initialized: now 1786553030039084 us; error 31 us; skew 500 ppm
I20260812 16:43:50.039486 21842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:50 dist-test-slave-f7th krb5kdc[21829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553030, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:50.047701 21842 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:50.049376 21842 webserver.cc:460] Webserver started at http://127.31.61.190:46129/ using document root <none> and password file <none>
I20260812 16:43:50.050076 21842 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:50.050184 21842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:50.050447 21842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:50.053632 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "101ffcf90f7347c8940b2da7b53ee793"
format_stamp: "Formatted at 2026-08-12 16:43:50 on dist-test-slave-f7th"
server_key: "39603650781059cd40f15d6d5b3e6ae1"
server_key_iv: "58e32e998e58a0b9940e2d27e24c44e7"
server_key_version: "encryptionkey@0"
I20260812 16:43:50.054397 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "101ffcf90f7347c8940b2da7b53ee793"
format_stamp: "Formatted at 2026-08-12 16:43:50 on dist-test-slave-f7th"
server_key: "39603650781059cd40f15d6d5b3e6ae1"
server_key_iv: "58e32e998e58a0b9940e2d27e24c44e7"
server_key_version: "encryptionkey@0"
I20260812 16:43:50.059836 21842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260812 16:43:50.063344 21859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:50.064754 21842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:43:50.064921 21842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "101ffcf90f7347c8940b2da7b53ee793"
format_stamp: "Formatted at 2026-08-12 16:43:50 on dist-test-slave-f7th"
server_key: "39603650781059cd40f15d6d5b3e6ae1"
server_key_iv: "58e32e998e58a0b9940e2d27e24c44e7"
server_key_version: "encryptionkey@0"
I20260812 16:43:50.065130 21842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:50.071020 21842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:50.075503 21842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:50.075836 21842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:50.094170 21842 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36227
I20260812 16:43:50.094211 21911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36227 every 8 connection(s)
I20260812 16:43:50.095562 21842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:50.099330 21912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:50.103567 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 21842
I20260812 16:43:50.103768 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:50.104183 31990 external_mini_cluster.cc:1428] Setting key 134a1c7a523a73e76adb7747711440cb
I20260812 16:43:50.112033 21912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793: Bootstrap starting.
I20260812 16:43:50.113969 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:50.106062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33773 (local address 127.31.61.190:36227)
0812 16:43:50.106940 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:50.106956 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:50.107008 (+ 52us) server_negotiation.cc:407] Connection header received
0812 16:43:50.109959 (+ 2951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:50.109995 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:50.110469 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:50.110936 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:50.111979 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:50.111988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:50.112005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:50.112029 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:50.112102 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:50.112133 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:50.112355 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:50.112498 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 16:43:50.112716 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":367,"threads_started":1}
I20260812 16:43:50.115984 21912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:50.117637 21912 log.cc:824] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:50.118252 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:50.120986 21912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793: No bootstrap required, opened a new log
I20260812 16:43:50.125469 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:50.122868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33693 (local address 127.31.61.190:36227)
0812 16:43:50.123063 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:50.123072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:50.123484 (+ 412us) server_negotiation.cc:407] Connection header received
0812 16:43:50.123692 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:50.123696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:50.123748 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:50.123835 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:50.124781 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:50.124789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:50.124793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:50.124809 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:50.124853 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:50.124868 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:50.124980 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:50.125091 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 16:43:50.125160 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 16:43:50.129732 21912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101ffcf90f7347c8940b2da7b53ee793" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36227 } }
I20260812 16:43:50.130102 21912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:50.130203 21912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101ffcf90f7347c8940b2da7b53ee793, State: Initialized, Role: FOLLOWER
I20260812 16:43:50.130813 21912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101ffcf90f7347c8940b2da7b53ee793" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36227 } }
I20260812 16:43:50.131060 21912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:50.131210 21912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:50.131381 21912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:50.133730 21912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101ffcf90f7347c8940b2da7b53ee793" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36227 } }
I20260812 16:43:50.134240 21912 leader_election.cc:302] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 101ffcf90f7347c8940b2da7b53ee793; no voters:
I20260812 16:43:50.134860 21912 leader_election.cc:288] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:50.135010 21922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:50.135649 21922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [term 1 LEADER]: Becoming Leader. State: Replica: 101ffcf90f7347c8940b2da7b53ee793, State: Running, Role: LEADER
I20260812 16:43:50.136207 21922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101ffcf90f7347c8940b2da7b53ee793" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36227 } }
I20260812 16:43:50.136950 21912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:50.138829 21924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 101ffcf90f7347c8940b2da7b53ee793. Latest consensus state: current_term: 1 leader_uuid: "101ffcf90f7347c8940b2da7b53ee793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101ffcf90f7347c8940b2da7b53ee793" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36227 } } }
I20260812 16:43:50.139169 21923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "101ffcf90f7347c8940b2da7b53ee793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101ffcf90f7347c8940b2da7b53ee793" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36227 } } }
I20260812 16:43:50.139401 21924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:50.139479 21923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:50.140451 21927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:50.146543 21927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:50.155210 21927 catalog_manager.cc:1347] Generated new cluster ID: ce72ca2027de43fea391e327ea3bc7c4
I20260812 16:43:50.155342 21927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:50.201779 21927 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:50.202234 21927 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:50.213570 21927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 101ffcf90f7347c8940b2da7b53ee793: Generated new TSK 0
I20260812 16:43:50.214454 21927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:50.224206 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21842
W20260812 16:43:50.225373 21907 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33773
W20260812 16:43:50.226195 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21842: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:50.226344 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 21842
2026-08-12T16:43:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:50 dist-test-slave-f7th krb5kdc[21943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:50 dist-test-slave-f7th krb5kdc[21943](info): set up 2 sockets
Aug 12 16:43:50 dist-test-slave-f7th krb5kdc[21943](info): commencing operation
krb5kdc: starting...
W20260812 16:43:51.291808 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:51 dist-test-slave-f7th krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:51.579841 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:35053
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:35109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:35053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:51.912137 21956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:51.912441 21956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:51.912587 21956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:51.919848 21956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:51.919970 21956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:51.920058 21956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:51.920137 21956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:51.920195 21956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:51.930244 21956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:35109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:35053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:35053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:51.931778 21956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:51.933152 21956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:51.944922 21962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:51.944904 21961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:51.945113 21964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:51.945623 21956 server_base.cc:1034] running on GCE node
I20260812 16:43:51.946386 21956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:51.947887 21956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:51.949124 21956 hybrid_clock.cc:648] HybridClock initialized: now 1786553031949066 us; error 37 us; skew 500 ppm
I20260812 16:43:51.949481 21956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:51 dist-test-slave-f7th krb5kdc[21943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553031, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:51.952696 21956 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:51.954211 21956 webserver.cc:460] Webserver started at http://127.31.61.190:40825/ using document root <none> and password file <none>
I20260812 16:43:51.954892 21956 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:51.955003 21956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:51.955304 21956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:51.958415 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "8095bd17d84341b3ad826a98f6ea1ce5"
format_stamp: "Formatted at 2026-08-12 16:43:51 on dist-test-slave-f7th"
server_key: "c406ae77883841a89777cd713c7a9dba"
server_key_iv: "218d71d96ae96ac760b6ac2a1c703da4"
server_key_version: "encryptionkey@0"
I20260812 16:43:51.959147 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "8095bd17d84341b3ad826a98f6ea1ce5"
format_stamp: "Formatted at 2026-08-12 16:43:51 on dist-test-slave-f7th"
server_key: "c406ae77883841a89777cd713c7a9dba"
server_key_iv: "218d71d96ae96ac760b6ac2a1c703da4"
server_key_version: "encryptionkey@0"
I20260812 16:43:51.964367 21956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 16:43:51.967912 21971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:51.969199 21956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:43:51.969348 21956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "8095bd17d84341b3ad826a98f6ea1ce5"
format_stamp: "Formatted at 2026-08-12 16:43:51 on dist-test-slave-f7th"
server_key: "c406ae77883841a89777cd713c7a9dba"
server_key_iv: "218d71d96ae96ac760b6ac2a1c703da4"
server_key_version: "encryptionkey@0"
I20260812 16:43:51.969527 21956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:51.992156 21956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:51.996757 21956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:51.997040 21956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:52.015878 21956 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:35053
I20260812 16:43:52.015897 22025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:35053 every 8 connection(s)
I20260812 16:43:52.017344 21956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:52.021158 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:52.026821 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 21956
I20260812 16:43:52.027002 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:52.027393 31990 external_mini_cluster.cc:1428] Setting key ee2c845da2126b82bd5de75b1650b790
I20260812 16:43:52.032130 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5: Bootstrap starting.
I20260812 16:43:52.035923 22026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:52.037207 22026 log.cc:824] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:52.040858 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5: No bootstrap required, opened a new log
Aug 12 16:43:52 dist-test-slave-f7th krb5kdc[21943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:43:52.049150 22026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095bd17d84341b3ad826a98f6ea1ce5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35053 } }
I20260812 16:43:52.049542 22026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:52.049649 22026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8095bd17d84341b3ad826a98f6ea1ce5, State: Initialized, Role: FOLLOWER
I20260812 16:43:52.050328 22026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095bd17d84341b3ad826a98f6ea1ce5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35053 } }
I20260812 16:43:52.050580 22026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:52.050722 22026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:52.050889 22026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:52.051333 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:52.029307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48823 (local address 127.31.61.190:35053)
0812 16:43:52.030168 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:52.030179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:52.030224 (+ 45us) server_negotiation.cc:407] Connection header received
0812 16:43:52.031906 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:52.031930 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:52.032351 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:52.032678 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:52.034451 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:52.035871 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:52.037569 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:52.038630 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:52.038769 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:52.043539 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:52.043547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:52.043563 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:52.043638 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:52.046743 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:52.047484 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:52.047491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:52.047498 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:52.047732 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:52.048164 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:52.048171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:52.048174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:52.048952 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:52.049236 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:52.049735 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 16:43:52.050172 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":395,"threads_started":1}
I20260812 16:43:52.053503 22026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095bd17d84341b3ad826a98f6ea1ce5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35053 } }
I20260812 16:43:52.053990 22026 leader_election.cc:302] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8095bd17d84341b3ad826a98f6ea1ce5; no voters:
I20260812 16:43:52.054637 22026 leader_election.cc:288] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:52.054771 22031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:52.056051 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:52.056142 22031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 8095bd17d84341b3ad826a98f6ea1ce5, State: Running, Role: LEADER
I20260812 16:43:52.056723 22031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095bd17d84341b3ad826a98f6ea1ce5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35053 } }
I20260812 16:43:52.056948 22026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:52.061347 22032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8095bd17d84341b3ad826a98f6ea1ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095bd17d84341b3ad826a98f6ea1ce5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35053 } } }
I20260812 16:43:52.061959 22032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:52.062166 22033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8095bd17d84341b3ad826a98f6ea1ce5. Latest consensus state: current_term: 1 leader_uuid: "8095bd17d84341b3ad826a98f6ea1ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8095bd17d84341b3ad826a98f6ea1ce5" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 35053 } } }
I20260812 16:43:52.062422 22033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5 [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:52.074509 22043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:52.080528 22043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:52.090368 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:52.061133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52155 (local address 127.31.61.190:35053)
0812 16:43:52.061310 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:52.061316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:52.071915 (+ 10599us) server_negotiation.cc:407] Connection header received
0812 16:43:52.072482 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:52.072489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:52.072585 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:52.072667 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:52.074807 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:52.075902 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:52.077400 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:52.078071 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:52.078195 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:52.081415 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:52.081425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:52.081430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:52.081487 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:52.085437 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:52.086361 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:52.086367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:52.086371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:52.086519 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:52.086969 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:52.086974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:52.086978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:52.087407 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:52.087670 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:52.089750 (+ 2080us) server_negotiation.cc:299] Negotiation successful
0812 16:43:52.090003 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:43:52.092190 22043 catalog_manager.cc:1347] Generated new cluster ID: 52d1c1f61ab74bf3a1e4024ada9144c7
I20260812 16:43:52.092502 22043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:52.097776 22054 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 16:43:52.111315 22043 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:52.112941 22043 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:52.121168 22043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8095bd17d84341b3ad826a98f6ea1ce5: Generated new TSK 0
I20260812 16:43:52.122354 22043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:52.135255 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 21956
I20260812 16:43:52.137087 22021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:43:52.230851 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 21956
2026-08-12T16:43:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:52 dist-test-slave-f7th krb5kdc[22058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:52 dist-test-slave-f7th krb5kdc[22058](info): set up 2 sockets
Aug 12 16:43:52 dist-test-slave-f7th krb5kdc[22058](info): commencing operation
krb5kdc: starting...
W20260812 16:43:53.275683 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:53 dist-test-slave-f7th krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:53.546581 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36097
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:53.882297 22071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:53.882580 22071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:53.882675 22071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:53.889948 22071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:53.890071 22071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:53.890125 22071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:53.890190 22071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:53.890234 22071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:53.900281 22071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:53.901588 22071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:53.902873 22071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:53.914573 22076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:53.914498 22077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:53.914939 22079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:53.915812 22071 server_base.cc:1034] running on GCE node
I20260812 16:43:53.916575 22071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:53.917977 22071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:53.919174 22071 hybrid_clock.cc:648] HybridClock initialized: now 1786553033919129 us; error 35 us; skew 500 ppm
I20260812 16:43:53.919505 22071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:53 dist-test-slave-f7th krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553033, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:53.922494 22071 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:53.924526 22071 webserver.cc:460] Webserver started at http://127.31.61.190:36407/ using document root <none> and password file <none>
I20260812 16:43:53.925228 22071 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:53.925343 22071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:53.925618 22071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:53.928609 22071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "248738acd87e4ca1847c3b3f9aeafe2d"
format_stamp: "Formatted at 2026-08-12 16:43:53 on dist-test-slave-f7th"
server_key: "fbdf6c3633393af7f319579d7faedaf5"
server_key_iv: "0af923638f762d1c92f35adb7e1a3ec2"
server_key_version: "encryptionkey@0"
I20260812 16:43:53.929323 22071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "248738acd87e4ca1847c3b3f9aeafe2d"
format_stamp: "Formatted at 2026-08-12 16:43:53 on dist-test-slave-f7th"
server_key: "fbdf6c3633393af7f319579d7faedaf5"
server_key_iv: "0af923638f762d1c92f35adb7e1a3ec2"
server_key_version: "encryptionkey@0"
I20260812 16:43:53.934515 22071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 16:43:53.937981 22086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:53.939383 22071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 16:43:53.939529 22071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "248738acd87e4ca1847c3b3f9aeafe2d"
format_stamp: "Formatted at 2026-08-12 16:43:53 on dist-test-slave-f7th"
server_key: "fbdf6c3633393af7f319579d7faedaf5"
server_key_iv: "0af923638f762d1c92f35adb7e1a3ec2"
server_key_version: "encryptionkey@0"
I20260812 16:43:53.939741 22071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:53.956462 22071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:53.960419 22071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:53.960666 22071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:53.978319 22071 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36097
I20260812 16:43:53.978343 22139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36097 every 8 connection(s)
I20260812 16:43:53.979682 22071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:53.983191 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 22071
I20260812 16:43:53.983335 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:53.983336 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:53.983820 31990 external_mini_cluster.cc:1428] Setting key d1f5461c191310ddd9337db75584f0df
I20260812 16:43:53.996977 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:53.985458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55661 (local address 127.31.61.190:36097)
0812 16:43:53.986254 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:53.986265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:53.986307 (+ 42us) server_negotiation.cc:407] Connection header received
0812 16:43:53.987915 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:53.987938 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:53.988326 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:53.988656 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:53.989254 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:53.990579 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:53.992402 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:53.993111 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:53.993254 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:53.994577 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:53.994586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:53.994601 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:53.995193 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:53.995279 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:53.995318 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:53.995415 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:53.995658 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 16:43:53.995929 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":357,"threads_started":1}
I20260812 16:43:53.997159 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d: Bootstrap starting.
I20260812 16:43:54.000855 22140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:54.000928 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:54.001878 22140 log.cc:824] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:54.005134 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d: No bootstrap required, opened a new log
I20260812 16:43:54.012001 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:54.005716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38911 (local address 127.31.61.190:36097)
0812 16:43:54.005910 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:54.005917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:54.006318 (+ 401us) server_negotiation.cc:407] Connection header received
0812 16:43:54.006469 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:54.006475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:54.006556 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:54.006671 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:54.007110 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:54.008255 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:54.009489 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:54.010232 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:54.010360 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:54.011258 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:54.011265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:54.011269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:54.011291 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:54.011336 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:54.011372 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:54.011464 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:54.011552 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 16:43:54.011671 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:43:54.014568 22140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248738acd87e4ca1847c3b3f9aeafe2d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36097 } }
I20260812 16:43:54.014938 22140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:54.015029 22140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248738acd87e4ca1847c3b3f9aeafe2d, State: Initialized, Role: FOLLOWER
I20260812 16:43:54.015561 22140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248738acd87e4ca1847c3b3f9aeafe2d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36097 } }
I20260812 16:43:54.015813 22140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:54.015969 22140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:54.016110 22140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:54.018586 22140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248738acd87e4ca1847c3b3f9aeafe2d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36097 } }
I20260812 16:43:54.019045 22140 leader_election.cc:302] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 248738acd87e4ca1847c3b3f9aeafe2d; no voters:
I20260812 16:43:54.019572 22140 leader_election.cc:288] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:54.019734 22151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:54.020407 22151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [term 1 LEADER]: Becoming Leader. State: Replica: 248738acd87e4ca1847c3b3f9aeafe2d, State: Running, Role: LEADER
I20260812 16:43:54.020946 22151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248738acd87e4ca1847c3b3f9aeafe2d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36097 } }
I20260812 16:43:54.021754 22140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:54.023746 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 248738acd87e4ca1847c3b3f9aeafe2d. Latest consensus state: current_term: 1 leader_uuid: "248738acd87e4ca1847c3b3f9aeafe2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248738acd87e4ca1847c3b3f9aeafe2d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36097 } } }
I20260812 16:43:54.024153 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:54.027653 22152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "248738acd87e4ca1847c3b3f9aeafe2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248738acd87e4ca1847c3b3f9aeafe2d" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36097 } } }
I20260812 16:43:54.028044 22152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:54.028218 22158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:54.033573 22158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:54.043295 22158 catalog_manager.cc:1347] Generated new cluster ID: bbd5b8950c224973a6b3b73270b6618f
I20260812 16:43:54.043404 22158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:54.056139 22158 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:54.057447 22158 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:54.068410 22158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 248738acd87e4ca1847c3b3f9aeafe2d: Generated new TSK 0
I20260812 16:43:54.069231 22158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:54.078739 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22071
W20260812 16:43:54.079890 22134 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55661
W20260812 16:43:54.080677 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22071: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:54.080811 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 22071
2026-08-12T16:43:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:54 dist-test-slave-f7th krb5kdc[22172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:54 dist-test-slave-f7th krb5kdc[22172](info): set up 2 sockets
Aug 12 16:43:54 dist-test-slave-f7th krb5kdc[22172](info): commencing operation
krb5kdc: starting...
W20260812 16:43:55.135911 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:55 dist-test-slave-f7th krb5kdc[22172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:55.424203 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36491
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:55.783445 22185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:55.783844 22185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:55.784017 22185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:55.791422 22185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:55.791554 22185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:55.791656 22185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:55.791738 22185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:55.791800 22185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:55.802019 22185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:55.803354 22185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:55.804736 22185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:55.816778 22190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:55.816843 22191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:55.816751 22193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:55.817407 22185 server_base.cc:1034] running on GCE node
I20260812 16:43:55.818203 22185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:55.819901 22185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:55.821131 22185 hybrid_clock.cc:648] HybridClock initialized: now 1786553035821085 us; error 27 us; skew 500 ppm
I20260812 16:43:55.821514 22185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:55 dist-test-slave-f7th krb5kdc[22172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:55.829803 22185 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:55.831437 22185 webserver.cc:460] Webserver started at http://127.31.61.190:42247/ using document root <none> and password file <none>
I20260812 16:43:55.832206 22185 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:55.832342 22185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:55.832680 22185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:55.836052 22185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "bd3fb48e4a214172b9b7b663edfd1abf"
format_stamp: "Formatted at 2026-08-12 16:43:55 on dist-test-slave-f7th"
server_key: "b653f47e27da1ba150c499ad337994fd"
server_key_iv: "5e4bab03d3238c1e9a6d799fb7ae68e4"
server_key_version: "encryptionkey@0"
I20260812 16:43:55.836825 22185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "bd3fb48e4a214172b9b7b663edfd1abf"
format_stamp: "Formatted at 2026-08-12 16:43:55 on dist-test-slave-f7th"
server_key: "b653f47e27da1ba150c499ad337994fd"
server_key_iv: "5e4bab03d3238c1e9a6d799fb7ae68e4"
server_key_version: "encryptionkey@0"
I20260812 16:43:55.842087 22185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 16:43:55.845721 22200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:55.847054 22185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 16:43:55.847218 22185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "bd3fb48e4a214172b9b7b663edfd1abf"
format_stamp: "Formatted at 2026-08-12 16:43:55 on dist-test-slave-f7th"
server_key: "b653f47e27da1ba150c499ad337994fd"
server_key_iv: "5e4bab03d3238c1e9a6d799fb7ae68e4"
server_key_version: "encryptionkey@0"
I20260812 16:43:55.847415 22185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:55.853211 22185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:55.857426 22185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:55.857724 22185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:55.876189 22185 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36491
I20260812 16:43:55.876221 22251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36491 every 8 connection(s)
I20260812 16:43:55.877583 22185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:55.881448 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 22185
I20260812 16:43:55.881599 22252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:55.881651 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:55.882171 31990 external_mini_cluster.cc:1428] Setting key 9c79de540df0318b7aeeb3871953bed7
I20260812 16:43:55.890053 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:55.884058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35847 (local address 127.31.61.190:36491)
0812 16:43:55.884917 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:55.884929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:55.884972 (+ 43us) server_negotiation.cc:407] Connection header received
0812 16:43:55.886561 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:55.886583 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:55.886925 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:55.887320 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:55.888457 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:55.888463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:55.888476 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:55.888497 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:55.888573 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:55.888594 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:55.888814 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:55.888971 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 16:43:55.889150 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":392,"threads_started":1}
I20260812 16:43:55.893747 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:55.895516 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf: Bootstrap starting.
I20260812 16:43:55.899130 22252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:55.900621 22252 log.cc:824] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:55.900733 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:55.897961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44715 (local address 127.31.61.190:36491)
0812 16:43:55.898159 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:55.898167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:55.898596 (+ 429us) server_negotiation.cc:407] Connection header received
0812 16:43:55.898735 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:55.898740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:55.898811 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:55.898936 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:55.899943 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:55.899950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:55.899956 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 16:43:55.899977 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:55.900032 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 16:43:55.900056 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:55.900194 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:55.900317 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 16:43:55.900400 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:43:55.903823 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf: No bootstrap required, opened a new log
I20260812 16:43:55.911446 22252 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3fb48e4a214172b9b7b663edfd1abf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36491 } }
I20260812 16:43:55.911885 22252 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 16:43:55.911979 22252 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3fb48e4a214172b9b7b663edfd1abf, State: Initialized, Role: FOLLOWER
I20260812 16:43:55.912516 22252 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3fb48e4a214172b9b7b663edfd1abf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36491 } }
I20260812 16:43:55.912738 22252 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 16:43:55.912863 22252 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 16:43:55.912992 22252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:55.915711 22252 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3fb48e4a214172b9b7b663edfd1abf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36491 } }
I20260812 16:43:55.916222 22252 leader_election.cc:302] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd3fb48e4a214172b9b7b663edfd1abf; no voters:
I20260812 16:43:55.916774 22252 leader_election.cc:288] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 16:43:55.916939 22262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:55.917572 22262 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [term 1 LEADER]: Becoming Leader. State: Replica: bd3fb48e4a214172b9b7b663edfd1abf, State: Running, Role: LEADER
I20260812 16:43:55.918128 22262 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3fb48e4a214172b9b7b663edfd1abf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36491 } }
I20260812 16:43:55.918764 22252 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:55.920985 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd3fb48e4a214172b9b7b663edfd1abf. Latest consensus state: current_term: 1 leader_uuid: "bd3fb48e4a214172b9b7b663edfd1abf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3fb48e4a214172b9b7b663edfd1abf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36491 } } }
I20260812 16:43:55.921433 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:55.921870 22263 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd3fb48e4a214172b9b7b663edfd1abf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3fb48e4a214172b9b7b663edfd1abf" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36491 } } }
I20260812 16:43:55.922135 22263 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:55.923419 22270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 16:43:55.928617 22270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:55.938592 22270 catalog_manager.cc:1347] Generated new cluster ID: 24a521fe4ca04261800e1ba6f57c261c
I20260812 16:43:55.938706 22270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:55.970800 22270 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:55.971268 22270 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:55.977663 22270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd3fb48e4a214172b9b7b663edfd1abf: Generated new TSK 0
I20260812 16:43:55.978695 22270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:43:55.993265 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22185
W20260812 16:43:55.994382 22247 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35847
W20260812 16:43:55.995136 31990 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22185: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 16:43:55.995276 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 22185
2026-08-12T16:43:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1914 ms)
[----------] 18 tests from AuthTokenIssuingTest (34431 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:43:56 dist-test-slave-f7th krb5kdc[22285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:43:56 dist-test-slave-f7th krb5kdc[22285](info): set up 2 sockets
Aug 12 16:43:56 dist-test-slave-f7th krb5kdc[22285](info): commencing operation
krb5kdc: starting...
W20260812 16:43:57.047878 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:43:57 dist-test-slave-f7th krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:43:57.326993 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:42667
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:42667,127.31.61.189:41985,127.31.61.188:37423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:43:57.663973 22298 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:43:57.664299 22298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:57.664404 22298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:57.664527 22298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:57.671841 22298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:57.671934 22298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:57.671995 22298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:57.672068 22298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:57.672145 22298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:57.682124 22298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:42667,127.31.61.189:41985,127.31.61.188:37423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:42667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:57.683496 22298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:57.684847 22298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:57.696789 22303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:57.696800 22304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:57.696797 22306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:57.697708 22298 server_base.cc:1034] running on GCE node
I20260812 16:43:57.698494 22298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:57.700013 22298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:57.701258 22298 hybrid_clock.cc:648] HybridClock initialized: now 1786553037701181 us; error 52 us; skew 500 ppm
I20260812 16:43:57.701620 22298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:57 dist-test-slave-f7th krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:57.704816 22298 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:43:57.706255 22298 webserver.cc:460] Webserver started at http://127.31.61.190:43267/ using document root <none> and password file <none>
I20260812 16:43:57.706945 22298 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:57.707046 22298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:57.707340 22298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:57.710336 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "722ac938933c49d7a747e71c1bbf800f"
format_stamp: "Formatted at 2026-08-12 16:43:57 on dist-test-slave-f7th"
server_key: "6491a1331726f6cab7dd7bbd7ffb7988"
server_key_iv: "f43e77bed27c53fc9808aced2c4a677a"
server_key_version: "encryptionkey@0"
I20260812 16:43:57.711134 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "722ac938933c49d7a747e71c1bbf800f"
format_stamp: "Formatted at 2026-08-12 16:43:57 on dist-test-slave-f7th"
server_key: "6491a1331726f6cab7dd7bbd7ffb7988"
server_key_iv: "f43e77bed27c53fc9808aced2c4a677a"
server_key_version: "encryptionkey@0"
I20260812 16:43:57.716420 22298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 16:43:57.720057 22313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:57.721472 22298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 16:43:57.721668 22298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "722ac938933c49d7a747e71c1bbf800f"
format_stamp: "Formatted at 2026-08-12 16:43:57 on dist-test-slave-f7th"
server_key: "6491a1331726f6cab7dd7bbd7ffb7988"
server_key_iv: "f43e77bed27c53fc9808aced2c4a677a"
server_key_version: "encryptionkey@0"
I20260812 16:43:57.721896 22298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:57.744874 22298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:57.749230 22298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:57.749496 22298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:57.769006 22298 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:42667
I20260812 16:43:57.769043 22365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:42667 every 8 connection(s)
I20260812 16:43:57.770460 22298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:43:57.774365 22366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:57.774719 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 22298
I20260812 16:43:57.774914 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:43:57.775561 31990 external_mini_cluster.cc:1428] Setting key 4ebb8b193d0cdce09df7519755d153a2
I20260812 16:43:57.781200 22366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:43:57 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.61.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.189@KRBTEST.COM" created.
I20260812 16:43:57.806631 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:57.784784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:42667 (local address 127.0.0.1:41756)
0812 16:43:57.785686 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:57.785729 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:57.788295 (+ 2566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:57.789241 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:57.789252 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:57.789968 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:57.790405 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:57.790428 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:57.792042 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:57.792047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:57.793606 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:57.793622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:57.794704 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:57.794710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:57.794889 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:57.795636 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:57.795673 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:57.798431 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:57.802119 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:57.802134 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:57.802147 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:57.802755 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:57.803245 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:57.803249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:57.803251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:57.803417 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:57.804508 (+ 1091us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:57.804515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:57.804718 (+ 203us) client_negotiation.cc:769] Sending connection context
0812 16:43:57.805008 (+ 290us) client_negotiation.cc:240] Negotiation successful
0812 16:43:57.805676 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":186,"thread_start_us":357,"threads_started":1}
I20260812 16:43:57.806690 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:57.784785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.31.61.190:42667)
0812 16:43:57.785648 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:57.785659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:57.785834 (+ 175us) server_negotiation.cc:407] Connection header received
0812 16:43:57.788744 (+ 2910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:57.788751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:57.789005 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:57.789174 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:57.790621 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:57.791873 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:57.793791 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:57.794449 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:57.794606 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:57.798709 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:57.798719 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:57.798733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:57.798776 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:57.801878 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:57.802923 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:57.802933 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:57.802936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:57.803125 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:57.803553 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:57.803557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:57.803560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:57.804356 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:57.804540 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:57.805141 (+ 601us) server_negotiation.cc:299] Negotiation successful
0812 16:43:57.805414 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":450,"threads_started":1}
I20260812 16:43:57.809607 22366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:43:57.810711 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:57.810287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38618)
0812 16:43:57.810461 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:57.810579 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 16:43:57.810993 22314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
I20260812 16:43:57.812599 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:57.812202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38626)
0812 16:43:57.812359 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:57.812443 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 16:43:57.812796 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:41985: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab.
Entry for principal kudu/127.31.61.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab.
I20260812 16:43:57.839154 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.189:41985
--webserver_interface=127.31.61.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab
--principal=kudu/127.31.61.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:42667,127.31.61.189:41985,127.31.61.188:37423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:43:57.862159 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } attempt: 1
I20260812 16:43:57.863447 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:57.862941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38634)
0812 16:43:57.863139 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:57.863262 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 16:43:57.864506 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:57.864138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38648)
0812 16:43:57.864285 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:57.864372 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 16:43:57.864687 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:41985: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
I20260812 16:43:57.932945 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } attempt: 2
I20260812 16:43:57.934197 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:57.933693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38650)
0812 16:43:57.933963 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:57.934056 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 16:43:57.935249 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:57.934883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38654)
0812 16:43:57.935045 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:57.935116 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 16:43:57.935437 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:41985: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
I20260812 16:43:58.026721 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } attempt: 3
I20260812 16:43:58.028559 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.027932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38670)
0812 16:43:58.028204 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.028313 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260812 16:43:58.030211 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.029646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38686)
0812 16:43:58.029876 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.030014 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260812 16:43:58.030471 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:41985: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
I20260812 16:43:58.173796 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } attempt: 4
I20260812 16:43:58.175436 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.174617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38694)
0812 16:43:58.175078 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.175216 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260812 16:43:58.177054 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.176514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38702)
0812 16:43:58.176739 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.176880 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260812 16:43:58.177301 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:41985: Network error: Client connection negotiation failed: client connection to 127.31.61.189:41985: connect: Connection refused (error 111)
W20260812 16:43:58.240832 22373 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:43:58.241420 22373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:58.241588 22373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:58.241742 22373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:58.252054 22373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:58.252214 22373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:58.252316 22373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:58.252409 22373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:58.252542 22373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:58.264945 22373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:42667,127.31.61.189:41985,127.31.61.188:37423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab
--principal=kudu/127.31.61.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.189:41985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.61.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:58.266861 22373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:58.269146 22373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:58.286474 22378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:58.286720 22381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:58.286638 22379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:58.288372 22373 server_base.cc:1034] running on GCE node
I20260812 16:43:58.289568 22373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:58.291828 22373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:58.293169 22373 hybrid_clock.cc:648] HybridClock initialized: now 1786553038293061 us; error 77 us; skew 500 ppm
I20260812 16:43:58.293710 22373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:58 dist-test-slave-f7th krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553038, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:58.298375 22373 init.cc:377] Logged in from keytab as kudu/127.31.61.189@KRBTEST.COM (short username kudu)
I20260812 16:43:58.300925 22373 webserver.cc:460] Webserver started at http://127.31.61.189:34259/ using document root <none> and password file <none>
I20260812 16:43:58.301944 22373 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:58.302052 22373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:58.302400 22373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:58.307566 22373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/data/instance:
uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff"
format_stamp: "Formatted at 2026-08-12 16:43:58 on dist-test-slave-f7th"
server_key: "058c57cb89fb3ec92b5de9143c439939"
server_key_iv: "608a2d6d606a6281b4bd48b80b27e950"
server_key_version: "encryptionkey@0"
I20260812 16:43:58.309139 22373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/wal/instance:
uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff"
format_stamp: "Formatted at 2026-08-12 16:43:58 on dist-test-slave-f7th"
server_key: "058c57cb89fb3ec92b5de9143c439939"
server_key_iv: "608a2d6d606a6281b4bd48b80b27e950"
server_key_version: "encryptionkey@0"
I20260812 16:43:58.320762 22373 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.005s
I20260812 16:43:58.326972 22388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:58.329125 22373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 16:43:58.329360 22373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/wal
uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff"
format_stamp: "Formatted at 2026-08-12 16:43:58 on dist-test-slave-f7th"
server_key: "058c57cb89fb3ec92b5de9143c439939"
server_key_iv: "608a2d6d606a6281b4bd48b80b27e950"
server_key_version: "encryptionkey@0"
I20260812 16:43:58.329768 22373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:58.353299 22373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:58.360461 22373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:58.360832 22373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:58.385136 22373 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.189:41985
I20260812 16:43:58.385161 22440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.189:41985 every 8 connection(s)
I20260812 16:43:58.386943 22373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
I20260812 16:43:58.389654 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 22373
I20260812 16:43:58.389932 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-1/wal/instance
I20260812 16:43:58.390765 31990 external_mini_cluster.cc:1428] Setting key 2fa67de1a3d114e30177c33e1669b313
I20260812 16:43:58.394027 22441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:58.406464 22441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:43:58 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553038, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.61.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.188@KRBTEST.COM" created.
I20260812 16:43:58.438695 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.410370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.31.61.190:42667)
0812 16:43:58.411142 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:58.411148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:58.412556 (+ 1408us) server_negotiation.cc:407] Connection header received
0812 16:43:58.415308 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:58.415334 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:58.415482 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:58.415723 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:58.417879 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.419377 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.422531 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.423562 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.423740 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:58.428739 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:58.428748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:58.428752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:58.428815 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:58.433510 (+ 4695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:58.434821 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:58.434826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:58.434830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:58.435028 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:58.435783 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:58.435788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:58.435791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:58.436337 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:58.436533 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:58.437760 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0812 16:43:58.438160 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":313,"threads_started":1}
I20260812 16:43:58.439865 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.411203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:42667 (local address 127.0.0.1:41772)
0812 16:43:58.412274 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.412345 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:58.414888 (+ 2543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:58.416018 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:58.416031 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:58.416797 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:58.417560 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:58.417584 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.419684 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.419689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:58.422247 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:58.422272 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.424100 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.424107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:58.424407 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:58.425337 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:58.425386 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:58.428426 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:58.433766 (+ 5340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:58.433778 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:58.433804 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:58.434620 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:58.435221 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:58.435226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:58.435228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:58.435580 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:58.436578 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:58.436585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:58.437138 (+ 553us) client_negotiation.cc:769] Sending connection context
0812 16:43:58.437537 (+ 399us) client_negotiation.cc:240] Negotiation successful
0812 16:43:58.438269 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":436,"threads_started":1}
I20260812 16:43:58.442034 22441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:43:58 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553038, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:43:58.464998 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.442942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38718)
0812 16:43:58.443228 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.443256 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:58.443435 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:58.445146 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:58.445155 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:58.445692 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:58.445891 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:58.445901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.448775 (+ 2874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.448782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:58.449816 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:58.449829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.451167 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.451179 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:58.451440 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:58.452492 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:58.452533 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:58.455474 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:58.461262 (+ 5788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:58.461266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:58.461269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:58.461795 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:58.462437 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:58.462441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:58.462448 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:58.462754 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:58.463973 (+ 1219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:58.463981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:58.464157 (+ 176us) client_negotiation.cc:769] Sending connection context
0812 16:43:58.464350 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 16:43:58.464656 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 16:43:58.465514 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.443339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.31.61.189:41985)
0812 16:43:58.444302 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:58.444320 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:58.444368 (+ 48us) server_negotiation.cc:407] Connection header received
0812 16:43:58.444444 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:58.444450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:58.444899 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:58.445168 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:58.446171 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.448564 (+ 2393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.449963 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.450971 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.451129 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:58.455744 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:58.455759 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:58.455779 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:58.455819 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:58.461067 (+ 5248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:58.461973 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:58.461983 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:58.461986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:58.462219 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:58.463014 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:58.463019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:58.463023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:58.463773 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:58.464079 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:58.464612 (+ 533us) server_negotiation.cc:299] Negotiation successful
0812 16:43:58.464999 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":523,"threads_started":1}
I20260812 16:43:58.468866 22441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:43:58.470345 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.469750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47624)
0812 16:43:58.469944 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.470116 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 16:43:58.470831 22389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.472971 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.472551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47628)
0812 16:43:58.472735 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.472832 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260812 16:43:58.473241 22441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.476570 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } attempt: 5
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab.
Entry for principal kudu/127.31.61.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab.
I20260812 16:43:58.480413 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.188:37423
--webserver_interface=127.31.61.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab
--principal=kudu/127.31.61.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:42667,127.31.61.189:41985,127.31.61.188:37423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 16:43:58 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:43:58.493096 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.477465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38720)
0812 16:43:58.477705 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.477730 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:58.477875 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:58.478475 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:58.478506 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:58.478946 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:58.479235 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:58.479259 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.480609 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.480614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:58.481695 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:58.481716 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.482739 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.482748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:58.483031 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:58.483801 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:58.483819 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:58.485957 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:58.489594 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:58.489607 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:58.489612 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:58.490295 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:58.490844 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:58.490850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:58.490854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:58.491057 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:58.491767 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:58.491777 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:58.492006 (+ 229us) client_negotiation.cc:769] Sending connection context
0812 16:43:58.492163 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 16:43:58.492571 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 16:43:58.493808 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.477690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.31.61.189:41985)
0812 16:43:58.477934 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:58.477945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:58.477992 (+ 47us) server_negotiation.cc:407] Connection header received
0812 16:43:58.478096 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:58.478129 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:58.478262 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:58.478456 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:58.479427 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.480432 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.481865 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:58.482543 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:58.482660 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:58.486139 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:58.486143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:58.486146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:58.486184 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:58.489427 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:58.490431 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:58.490437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:58.490441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:58.490641 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:58.491197 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:58.491202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:58.491204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:58.491574 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:58.491811 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:58.492810 (+ 999us) server_negotiation.cc:299] Negotiation successful
0812 16:43:58.493128 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 16:43:58.495060 22366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:43:58.496423 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.495818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47640)
0812 16:43:58.496035 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.496143 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 16:43:58.497978 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.497447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47650)
0812 16:43:58.497711 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.497824 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260812 16:43:58.498229 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.522684 22441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 1
I20260812 16:43:58.524156 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.523637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47666)
0812 16:43:58.523867 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.523979 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 16:43:58.525609 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.524950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47676)
0812 16:43:58.525248 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.525399 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 16:43:58.525900 22441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.549540 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 1
I20260812 16:43:58.550956 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.550353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47682)
0812 16:43:58.550652 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.550761 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260812 16:43:58.552296 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.551846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47692)
0812 16:43:58.552058 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.552161 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260812 16:43:58.552580 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.594194 22441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 2
I20260812 16:43:58.595470 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.594965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47706)
0812 16:43:58.595164 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.595262 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 16:43:58.596740 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.596361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47722)
0812 16:43:58.596534 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.596610 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 16:43:58.596956 22441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.620843 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 2
I20260812 16:43:58.622325 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.621754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47732)
0812 16:43:58.622022 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.622112 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 16:43:58.623649 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.623123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47734)
0812 16:43:58.623324 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.623433 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260812 16:43:58.623921 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.688210 22441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 3
I20260812 16:43:58.689666 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.689064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47736)
0812 16:43:58.689315 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.689462 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 16:43:58.691262 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.690696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47748)
0812 16:43:58.691019 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.691120 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
W20260812 16:43:58.691530 22441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.730218 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 3
I20260812 16:43:58.731815 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.731051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47750)
0812 16:43:58.731331 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.731500 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 16:43:58.733338 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.732814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47760)
0812 16:43:58.733049 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.733202 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260812 16:43:58.733798 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.834949 22441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 4
I20260812 16:43:58.837518 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.836499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47768)
0812 16:43:58.836859 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.837162 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260812 16:43:58.838922 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.838499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47780)
0812 16:43:58.838679 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.838774 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 16:43:58.839226 22441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
I20260812 16:43:58.911103 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 4
I20260812 16:43:58.913028 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.912263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47794)
0812 16:43:58.912584 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.912766 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260812 16:43:58.914287 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:58.913826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47806)
0812 16:43:58.914015 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:58.914142 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 16:43:58.914574 22366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:37423: Network error: Client connection negotiation failed: client connection to 127.31.61.188:37423: connect: Connection refused (error 111)
W20260812 16:43:58.940238 22449 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:43:58.940711 22449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:43:58.940819 22449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:43:58.940986 22449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:43:58.951304 22449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:43:58.951444 22449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:43:58.951570 22449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:43:58.951658 22449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:43:58.951766 22449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:43:58.965535 22449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:42667,127.31.61.189:41985,127.31.61.188:37423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab
--principal=kudu/127.31.61.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.188:37423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.61.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:43:58.967445 22449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:43:58.969272 22449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:43:58.985399 22454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:58.985918 22457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:43:58.985999 22455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:43:58.989593 22449 server_base.cc:1034] running on GCE node
I20260812 16:43:58.990752 22449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:43:58.993937 22449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:43:58.997432 22449 hybrid_clock.cc:648] HybridClock initialized: now 1786553038997235 us; error 152 us; skew 500 ppm
I20260812 16:43:58.998008 22449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:43:59 dist-test-slave-f7th krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:43:59.004966 22449 init.cc:377] Logged in from keytab as kudu/127.31.61.188@KRBTEST.COM (short username kudu)
I20260812 16:43:59.007498 22449 webserver.cc:460] Webserver started at http://127.31.61.188:41071/ using document root <none> and password file <none>
I20260812 16:43:59.008550 22449 fs_manager.cc:359] Metadata directory not provided
I20260812 16:43:59.008720 22449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:43:59.009260 22449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:43:59.016332 22449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/data/instance:
uuid: "eb95ae495b7546a6a62decd2ead45c0b"
format_stamp: "Formatted at 2026-08-12 16:43:59 on dist-test-slave-f7th"
server_key: "8e0df98297bb7cd16d27c7e0133bdcd9"
server_key_iv: "1cc1bb071d9000540e3a838d8d01b035"
server_key_version: "encryptionkey@0"
I20260812 16:43:59.018514 22449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/wal/instance:
uuid: "eb95ae495b7546a6a62decd2ead45c0b"
format_stamp: "Formatted at 2026-08-12 16:43:59 on dist-test-slave-f7th"
server_key: "8e0df98297bb7cd16d27c7e0133bdcd9"
server_key_iv: "1cc1bb071d9000540e3a838d8d01b035"
server_key_version: "encryptionkey@0"
I20260812 16:43:59.028469 22449 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.004s
I20260812 16:43:59.034106 22465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:43:59.036492 22449 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 16:43:59.036746 22449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/wal
uuid: "eb95ae495b7546a6a62decd2ead45c0b"
format_stamp: "Formatted at 2026-08-12 16:43:59 on dist-test-slave-f7th"
server_key: "8e0df98297bb7cd16d27c7e0133bdcd9"
server_key_iv: "1cc1bb071d9000540e3a838d8d01b035"
server_key_version: "encryptionkey@0"
I20260812 16:43:59.037093 22449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:43:59.068854 22449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:43:59.075799 22449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:43:59.076246 22449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:43:59.101351 22449 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.188:37423
I20260812 16:43:59.101320 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.188:37423 every 8 connection(s)
I20260812 16:43:59.103219 22449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
I20260812 16:43:59.108174 22518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:43:59.113306 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 22449
I20260812 16:43:59.113588 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786552676633950-31990-0/minicluster-data/master-2/wal/instance
I20260812 16:43:59.114290 31990 external_mini_cluster.cc:1428] Setting key a427d3a8bd9156fb470dedca3911f6f3
I20260812 16:43:59.118769 22518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:43:59.138546 22441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 5
Aug 12 16:43:59 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
Aug 12 16:43:59 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
Aug 12 16:43:59 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553038, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
I20260812 16:43:59.153183 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.124808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.31.61.190:42667)
0812 16:43:59.127193 (+ 2385us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.127208 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.127903 (+ 695us) server_negotiation.cc:407] Connection header received
0812 16:43:59.127980 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.128004 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.128094 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.128210 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.130753 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.132230 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.135083 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.136695 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.137202 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.143752 (+ 6550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.143770 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.143782 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.143882 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.148068 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.149558 (+ 1490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.149579 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.149582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.149827 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.150501 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.150507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.150518 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.151012 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.151160 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.152079 (+ 919us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.152436 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2191,"thread_start_us":1990,"threads_started":1}
I20260812 16:43:59.154369 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.124688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:42667 (local address 127.0.0.1:41786)
0812 16:43:59.127411 (+ 2723us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:59.127501 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:59.127764 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:59.128379 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:59.128395 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:59.129951 (+ 1556us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:59.130514 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.130535 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.132463 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.132477 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.134671 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.134696 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.137660 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.137670 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.137929 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.139061 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:59.139108 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:59.143289 (+ 4181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:59.148193 (+ 4904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.148206 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.148214 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.149380 (+ 1166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.150029 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.150041 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.150045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.150354 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.151264 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:59.151273 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:59.151533 (+ 260us) client_negotiation.cc:769] Sending connection context
0812 16:43:59.151909 (+ 376us) client_negotiation.cc:240] Negotiation successful
0812 16:43:59.152490 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2318,"thread_start_us":1735,"threads_started":1}
I20260812 16:43:59.154438 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.117116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.31.61.188:37423)
0812 16:43:59.118090 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.118118 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.118184 (+ 66us) server_negotiation.cc:407] Connection header received
0812 16:43:59.120346 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.120387 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.121278 (+ 891us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.122023 (+ 745us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.124090 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.127407 (+ 3317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.130322 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.131989 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.132326 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.141387 (+ 9061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.141409 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.141447 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.141520 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.146801 (+ 5281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.148055 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.148065 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.148076 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.148311 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.148960 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.148967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.148971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.149898 (+ 927us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.150333 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.151266 (+ 933us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.152091 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":425,"threads_started":1}
I20260812 16:43:59.161641 22518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:43:59.161916 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.139719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.31.61.188:37423)
0812 16:43:59.141138 (+ 1419us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.141144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.141169 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:43:59.141449 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.141460 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.141696 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.141850 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.143241 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.145342 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.147435 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.148221 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.148389 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.153848 (+ 5459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.153881 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.153895 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.153960 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.158049 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.159224 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.159230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.159234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.159418 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.159946 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.159952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.159955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.160444 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.160629 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.161154 (+ 525us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.161417 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1242,"thread_start_us":927,"threads_started":1}
I20260812 16:43:59.161964 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.139478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47830)
0812 16:43:59.140024 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:59.140060 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:59.140274 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:59.141942 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:59.141950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:59.142621 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:59.142967 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.142982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.145516 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.145523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.147286 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.147302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.148441 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.148455 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.148773 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.149478 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:59.149525 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:59.153421 (+ 3896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:59.158289 (+ 4868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.158295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.158298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.159077 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.159573 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.159633 (+ 60us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.159636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.159798 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.160582 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:59.160595 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:59.160795 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 16:43:59.160988 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 16:43:59.161418 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349}
I20260812 16:43:59.171905 22441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff: Bootstrap starting.
Aug 12 16:43:59 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
Aug 12 16:43:59 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:43:59.178834 22441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:59.181262 22441 log.cc:824] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:59.188594 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.162543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:41985 (local address 127.0.0.1:38736)
0812 16:43:59.162861 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:59.162899 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:59.163049 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:59.165586 (+ 2537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:59.165598 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:59.166362 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:59.166708 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.166727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.168924 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.168930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.170584 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.170603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.172334 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.172348 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.172685 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.173679 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:59.173702 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:59.178108 (+ 4406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:59.183497 (+ 5389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.183508 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.183511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.184550 (+ 1039us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.185306 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.185314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.185318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.185668 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.186729 (+ 1061us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:59.186736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:59.187018 (+ 282us) client_negotiation.cc:769] Sending connection context
0812 16:43:59.187292 (+ 274us) client_negotiation.cc:240] Negotiation successful
0812 16:43:59.187886 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260812 16:43:59.188638 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.162548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.31.61.189:41985)
0812 16:43:59.165052 (+ 2504us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.165061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.165096 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:43:59.165182 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.165194 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.165329 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.165505 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.167053 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.168728 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.170843 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.172110 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.172304 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.178543 (+ 6239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.178552 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.178556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.178629 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.183243 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.184764 (+ 1521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.184779 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.184782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.185054 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.185877 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.185886 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.185889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.186563 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.186795 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.187490 (+ 695us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.187908 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2292,"thread_start_us":2044,"threads_started":1}
I20260812 16:43:59.188673 22441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff: No bootstrap required, opened a new log
I20260812 16:43:59.190927 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.161029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.31.61.190:42667)
0812 16:43:59.161256 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.161271 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.161722 (+ 451us) server_negotiation.cc:407] Connection header received
0812 16:43:59.161785 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.161791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.161886 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.162016 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.164339 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.165881 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.168328 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.169411 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.169718 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.177333 (+ 7615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.177350 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.177355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.177438 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.184790 (+ 7352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.186015 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.186034 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.186039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.186331 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.187130 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.187136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.187139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.187832 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.188304 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.190130 (+ 1826us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.190492 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 16:43:59.190990 22518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:43:59.191829 22366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } attempt: 5
Aug 12 16:43:59 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
I20260812 16:43:59.206871 22441 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } }
Aug 12 16:43:59 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
I20260812 16:43:59.208038 22441 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3693d3af824a5cb2e98a2a5ed6f7ff, State: Initialized, Role: FOLLOWER
I20260812 16:43:59.209057 22441 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } }
Aug 12 16:43:59 dist-test-slave-f7th krb5kdc[22285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:43:59.217022 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.219259 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.194403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.31.61.188:37423)
0812 16:43:59.194633 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.194638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.194659 (+ 21us) server_negotiation.cc:407] Connection header received
0812 16:43:59.194714 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.194718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.194811 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.194973 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.196610 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.198243 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.200698 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.201639 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.201842 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.208260 (+ 6418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.208268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.208281 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.208349 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.214243 (+ 5894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.215376 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.215382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.215394 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.215686 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.216311 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.216326 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.216335 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.217045 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.217264 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.218144 (+ 880us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.218552 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 16:43:59.220947 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.192826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47850)
0812 16:43:59.193426 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:59.193458 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:59.193621 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:59.195223 (+ 1602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:59.195229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:59.196051 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:59.196367 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.196387 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.198500 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.198511 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.200484 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.200507 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.202209 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.202217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.202532 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.203497 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:59.203546 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:59.207940 (+ 4394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:59.214526 (+ 6586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.214532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.214543 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.215231 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.215911 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.215918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.215921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.216133 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.217400 (+ 1267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:59.217406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:59.217687 (+ 281us) client_negotiation.cc:769] Sending connection context
0812 16:43:59.219793 (+ 2106us) client_negotiation.cc:240] Negotiation successful
0812 16:43:59.220286 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425}
I20260812 16:43:59.221518 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.192686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:37423 (local address 127.0.0.1:47834)
0812 16:43:59.193494 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0812 16:43:59.193523 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 16:43:59.193671 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:43:59.195886 (+ 2215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:43:59.195891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:43:59.196617 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:43:59.196944 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.196971 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.199000 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.199014 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.201153 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:43:59.201177 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.203440 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.203452 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:43:59.203846 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.204730 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:43:59.204779 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:43:59.209028 (+ 4249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:43:59.216327 (+ 7299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.216345 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.216348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.217280 (+ 932us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.218032 (+ 752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:43:59.218040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:43:59.218043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:43:59.218368 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:43:59.219664 (+ 1296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:43:59.219734 (+ 70us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:43:59.220038 (+ 304us) client_negotiation.cc:769] Sending connection context
0812 16:43:59.220241 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 16:43:59.220742 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614}
I20260812 16:43:59.221877 22441 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:59.222101 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:43:59.222971 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.193996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.31.61.188:37423)
0812 16:43:59.194264 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.194273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.194308 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:43:59.194385 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.194397 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.194539 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.194674 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.197243 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.198674 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.201443 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.202500 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.202757 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.209666 (+ 6909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.209675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.209679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.209736 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.216075 (+ 6339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.217519 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.217531 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.217535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.217795 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.218607 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.218618 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.218621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.219259 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.219496 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.222069 (+ 2573us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.222423 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":96,"spinlock_wait_cycles":62336}
I20260812 16:43:59.231805 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.197912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.31.61.189:41985)
0812 16:43:59.198194 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.198200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.200097 (+ 1897us) server_negotiation.cc:407] Connection header received
0812 16:43:59.200487 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.200500 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.200639 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.200754 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.203889 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.205532 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.208352 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.209377 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.209480 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.219846 (+ 10366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.219868 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.219874 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.219965 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.226210 (+ 6245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.227421 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.227427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.227429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.227656 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.228258 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.228272 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.228282 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.229094 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.230831 (+ 1737us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.231004 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.231300 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 16:43:59.235875 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b: Bootstrap starting.
I20260812 16:43:59.240414 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:43:59.241312 22366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f: Bootstrap starting.
I20260812 16:43:59.241703 22518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:59.244252 22518 log.cc:824] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:59.249249 22366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f: Neither blocks nor log segments found. Creating new log.
I20260812 16:43:59.246685 22340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "722ac938933c49d7a747e71c1bbf800f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41790
I20260812 16:43:59.257107 22366 log.cc:824] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f: Log is configured to *not* fsync() on all Append() calls
I20260812 16:43:59.259389 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b: No bootstrap required, opened a new log
W20260812 16:43:59.265050 22538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:43:59.265278 22538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:43:59.254793 22373 master_runner.cc:186] Error getting master registration for 127.31.61.190:42667: OK, instance_id { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" instance_seqno: 1786553037763005 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:43:59.266510 22373 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 16:43:59.269351 22366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f: No bootstrap required, opened a new log
I20260812 16:43:59.275470 22518 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } }
I20260812 16:43:59.276026 22518 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb95ae495b7546a6a62decd2ead45c0b, State: Initialized, Role: FOLLOWER
I20260812 16:43:59.276573 22518 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } }
I20260812 16:43:59.277606 22366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } }
I20260812 16:43:59.278052 22366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722ac938933c49d7a747e71c1bbf800f, State: Initialized, Role: FOLLOWER
I20260812 16:43:59.277899 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.278553 22366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } }
I20260812 16:43:59.278743 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:43:59.279129 22518 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:59.285933 22543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.286371 22543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:43:59.287393 22366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:43:59.287622 22340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 16:43:59.287838 22340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:59.291509 22340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } }
I20260812 16:43:59.294531 22415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "722ac938933c49d7a747e71c1bbf800f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff"
I20260812 16:43:59.295065 22415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:59.298988 22415 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 722ac938933c49d7a747e71c1bbf800f in term 1.
I20260812 16:43:59.300721 22314 leader_election.cc:302] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722ac938933c49d7a747e71c1bbf800f, ef3693d3af824a5cb2e98a2a5ed6f7ff; no voters:
I20260812 16:43:59.301291 22543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:43:59.302248 22340 leader_election.cc:288] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [CANDIDATE]: Term 1 election: Requested vote from peers ef3693d3af824a5cb2e98a2a5ed6f7ff (127.31.61.189:41985), eb95ae495b7546a6a62decd2ead45c0b (127.31.61.188:37423)
I20260812 16:43:59.302623 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "722ac938933c49d7a747e71c1bbf800f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eb95ae495b7546a6a62decd2ead45c0b"
I20260812 16:43:59.303085 22492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:43:59.305461 22492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 722ac938933c49d7a747e71c1bbf800f in term 1.
I20260812 16:43:59.322580 22543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [term 1 LEADER]: Becoming Leader. State: Replica: 722ac938933c49d7a747e71c1bbf800f, State: Running, Role: LEADER
I20260812 16:43:59.323310 22543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } }
W20260812 16:43:59.329635 22471 tablet.cc:2369] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:59.339686 22552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722ac938933c49d7a747e71c1bbf800f. Latest consensus state: current_term: 1 leader_uuid: "722ac938933c49d7a747e71c1bbf800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.340196 22552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:59.349784 22564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260812 16:43:59.364383 22394 tablet.cc:2369] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 16:43:59.365463 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.313159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.31.61.190:42667)
0812 16:43:59.313346 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.313352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.328675 (+ 15323us) server_negotiation.cc:407] Connection header received
0812 16:43:59.329925 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.329931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.330014 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.330088 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.334046 (+ 3958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.335179 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.339087 (+ 3908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.339812 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.339918 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.349062 (+ 9144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.349070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.349075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.349123 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.358802 (+ 9679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.362158 (+ 3356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.362167 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.362170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.362360 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.363453 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.363459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.363462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.363869 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.364141 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.364790 (+ 649us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.365064 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 16:43:59.366052 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.310513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.31.61.189:41985)
0812 16:43:59.310733 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.310740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.328282 (+ 17542us) server_negotiation.cc:407] Connection header received
0812 16:43:59.328638 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.328645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.328741 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.328818 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.333137 (+ 4319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.334380 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.350837 (+ 16457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.351539 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.351669 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.356055 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.356062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.356067 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.356112 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.360848 (+ 4736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.362196 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.362202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.362205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.362360 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.362966 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.362972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.362979 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.363388 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.363659 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.365436 (+ 1777us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.365694 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:43:59.378930 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:43:59.329233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.31.61.188:37423)
0812 16:43:59.329459 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.329466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.338668 (+ 9202us) server_negotiation.cc:407] Connection header received
0812 16:43:59.340434 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.340440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.340531 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.340610 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.346120 (+ 5510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.347148 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.350834 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:43:59.351531 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:43:59.351669 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:43:59.370661 (+ 18992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:43:59.370670 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:43:59.370676 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:43:59.370729 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:43:59.374966 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.376133 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.376140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.376144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.376352 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:43:59.376853 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:43:59.376859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:43:59.376863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:43:59.377269 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:43:59.377578 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0812 16:43:59.378227 (+ 649us) server_negotiation.cc:299] Negotiation successful
0812 16:43:59.378525 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:43:59.394109 22564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:43:59.410971 22578 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 16:43:59.408499 22449 master_runner.cc:186] Error getting master registration for 127.31.61.190:42667: OK, instance_id { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" instance_seqno: 1786553037763005 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:43:59.418533 22449 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 16:43:59.424811 22492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [term 1 FOLLOWER]: Refusing update from remote peer 722ac938933c49d7a747e71c1bbf800f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:59.424865 22415 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [term 1 FOLLOWER]: Refusing update from remote peer 722ac938933c49d7a747e71c1bbf800f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:43:59.425931 22543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:43:59.426676 22543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 16:43:59.440053 22571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:43:59.440213 22571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:43:59.450599 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722ac938933c49d7a747e71c1bbf800f. Latest consensus state: current_term: 1 leader_uuid: "722ac938933c49d7a747e71c1bbf800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.450929 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:43:59.454169 22579 mvcc.cc:205] Tried to move back new op lower bound from 7317721249467490304 to 7317721249075113984. Current Snapshot: MvccSnapshot[applied={T|T < 7317721249467490304}]
I20260812 16:43:59.460069 22543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "722ac938933c49d7a747e71c1bbf800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.460417 22543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:59.460695 22581 mvcc.cc:205] Tried to move back new op lower bound from 7317721249467490304 to 7317721249075113984. Current Snapshot: MvccSnapshot[applied={T|T < 7317721249467490304}]
I20260812 16:43:59.463176 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "722ac938933c49d7a747e71c1bbf800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.463435 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:43:59.464965 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722ac938933c49d7a747e71c1bbf800f. Latest consensus state: current_term: 1 leader_uuid: "722ac938933c49d7a747e71c1bbf800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.465310 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:43:59.465879 22564 catalog_manager.cc:1347] Generated new cluster ID: 55f1753310ee4d15bf1f3e25402add5c
I20260812 16:43:59.465993 22564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:43:59.468609 22552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "722ac938933c49d7a747e71c1bbf800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.468873 22552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f [sys.catalog]: This master's current role is: LEADER
I20260812 16:43:59.480271 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "722ac938933c49d7a747e71c1bbf800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "722ac938933c49d7a747e71c1bbf800f" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 42667 } } peers { permanent_uuid: "ef3693d3af824a5cb2e98a2a5ed6f7ff" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 41985 } } peers { permanent_uuid: "eb95ae495b7546a6a62decd2ead45c0b" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 37423 } } }
I20260812 16:43:59.480597 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:43:59.490454 22564 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:43:59.492820 22564 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:43:59.519753 22564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 722ac938933c49d7a747e71c1bbf800f: Generated new TSK 0
I20260812 16:43:59.520756 22564 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 16:43:59.670825 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.648811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.31.61.190:42667)
0812 16:43:59.649045 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.649051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.666549 (+ 17498us) server_negotiation.cc:407] Connection header received
0812 16:43:59.669089 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.669096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.669201 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.669279 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.670566 (+ 1287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41814: BlockingRecv error: recv got EOF from 127.0.0.1:41814 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 16:43:59.673950 22524 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.667165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.31.61.188:37423)
0812 16:43:59.667354 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.667359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.672275 (+ 4916us) server_negotiation.cc:407] Connection header received
0812 16:43:59.672522 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.672528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.672615 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.672695 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.673660 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47866: BlockingRecv error: recv got EOF from 127.0.0.1:47866 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 16:43:59.674095 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.650836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.31.61.189:41985)
0812 16:43:59.651053 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.651059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.672201 (+ 21142us) server_negotiation.cc:407] Connection header received
0812 16:43:59.672477 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.672483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.672569 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.672655 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.673849 (+ 1194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38768: BlockingRecv error: recv got EOF from 127.0.0.1:38768 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 16:43:59.723348 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.710044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.31.61.189:41985)
0812 16:43:59.710267 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.710274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.720928 (+ 10654us) server_negotiation.cc:407] Connection header received
0812 16:43:59.721566 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.721573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.721718 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.721801 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.723051 (+ 1250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38772: BlockingRecv error: recv got EOF from 127.0.0.1:38772 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 16:43:59.773415 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.759945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.31.61.189:41985)
0812 16:43:59.760134 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.760141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.771100 (+ 10959us) server_negotiation.cc:407] Connection header received
0812 16:43:59.771652 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.771659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.771752 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.771818 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.773149 (+ 1331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38786: BlockingRecv error: recv got EOF from 127.0.0.1:38786 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 16:43:59.823915 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.810735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.31.61.189:41985)
0812 16:43:59.810921 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.810927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.821664 (+ 10737us) server_negotiation.cc:407] Connection header received
0812 16:43:59.822233 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.822241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.822350 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.822427 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.823623 (+ 1196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38788: BlockingRecv error: recv got EOF from 127.0.0.1:38788 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 16:43:59.844627 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.842051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.31.61.189:41985)
0812 16:43:59.842275 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.842281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.842874 (+ 593us) server_negotiation.cc:407] Connection header received
0812 16:43:59.843117 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.843124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.843225 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.843337 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.844365 (+ 1028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38802: BlockingRecv error: recv got EOF from 127.0.0.1:38802 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260812 16:43:59.859314 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.856772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.31.61.189:41985)
0812 16:43:59.856957 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.856963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.857492 (+ 529us) server_negotiation.cc:407] Connection header received
0812 16:43:59.857807 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.857814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.857927 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.858035 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.859068 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38812: BlockingRecv error: recv got EOF from 127.0.0.1:38812 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 16:43:59.882889 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.880251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.31.61.189:41985)
0812 16:43:59.880456 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.880462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.881024 (+ 562us) server_negotiation.cc:407] Connection header received
0812 16:43:59.881391 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.881399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.881502 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.881585 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.882643 (+ 1058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38828: BlockingRecv error: recv got EOF from 127.0.0.1:38828 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 16:43:59.922166 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.919434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.31.61.189:41985)
0812 16:43:59.919644 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.919650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.920212 (+ 562us) server_negotiation.cc:407] Connection header received
0812 16:43:59.920495 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.920502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.920632 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.920763 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.921818 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38838: BlockingRecv error: recv got EOF from 127.0.0.1:38838 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 16:43:59.992260 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:43:59.990449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.31.61.189:41985)
0812 16:43:59.990633 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 16:43:59.990638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:43:59.990976 (+ 338us) server_negotiation.cc:407] Connection header received
0812 16:43:59.991125 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:43:59.991129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:43:59.991210 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:43:59.991307 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:43:59.992040 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38848: BlockingRecv error: recv got EOF from 127.0.0.1:38848 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 16:44:00.126679 22525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:00.124812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.31.61.189:41985)
0812 16:44:00.125030 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:00.125036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:00.125264 (+ 228us) server_negotiation.cc:407] Connection header received
0812 16:44:00.125502 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:00.125507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:00.125584 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:00.125673 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:00.126442 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38852: BlockingRecv error: recv got EOF from 127.0.0.1:38852 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:44:00.267426 22538 catalog_manager.cc:1259] Loaded cluster ID: 55f1753310ee4d15bf1f3e25402add5c
I20260812 16:44:00.267550 22538 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff: loading cluster ID for follower catalog manager: success
I20260812 16:44:00.270078 22538 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff: acquiring CA information for follower catalog manager: success
I20260812 16:44:00.272032 22538 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef3693d3af824a5cb2e98a2a5ed6f7ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:44:00.395984 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:00.389387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.31.61.189:41985)
0812 16:44:00.389581 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:00.389588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:00.389866 (+ 278us) server_negotiation.cc:407] Connection header received
0812 16:44:00.390122 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:00.390127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:00.390286 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:00.390426 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:00.390898 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.391691 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.393644 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.394074 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.394187 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:00.394749 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:00.395333 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:00.395443 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:00.395572 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 16:44:00.395712 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 16:44:00.404348 22656 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:00.402247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47878 (local address 127.31.61.188:37423)
0812 16:44:00.402752 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:00.402758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:00.402917 (+ 159us) server_negotiation.cc:407] Connection header received
0812 16:44:00.403148 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:00.403152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:00.403265 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:00.403361 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:00.404167 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47878: BlockingRecv error: recv got EOF from 127.0.0.1:47878 (error 108)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":263,"threads_started":1}
I20260812 16:44:00.442274 22571 catalog_manager.cc:1259] Loaded cluster ID: 55f1753310ee4d15bf1f3e25402add5c
I20260812 16:44:00.442385 22571 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b: loading cluster ID for follower catalog manager: success
I20260812 16:44:00.445041 22571 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b: acquiring CA information for follower catalog manager: success
I20260812 16:44:00.446835 22571 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb95ae495b7546a6a62decd2ead45c0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:44:00.924868 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:00.919952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.31.61.189:41985)
0812 16:44:00.920445 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:00.920450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:00.920577 (+ 127us) server_negotiation.cc:407] Connection header received
0812 16:44:00.920708 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:00.920727 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:00.920803 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:00.920902 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:00.921305 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.922039 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.923433 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.923858 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.923948 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:00.924261 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:00.924358 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:00.924415 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:00.924519 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 16:44:00.924580 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
I20260812 16:44:00.936178 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:00.930662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.31.61.188:37423)
0812 16:44:00.931180 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:00.931195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:00.931213 (+ 18us) server_negotiation.cc:407] Connection header received
0812 16:44:00.931277 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:00.931281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:00.931446 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:00.931555 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:00.931999 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.932683 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.933989 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.934392 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.934513 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:00.935011 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:00.935548 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:00.935741 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:00.935805 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 16:44:00.935888 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":209,"threads_started":1}
I20260812 16:44:00.950908 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:00.942406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.31.61.189:41985)
0812 16:44:00.942772 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:00.942779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:00.944817 (+ 2038us) server_negotiation.cc:407] Connection header received
0812 16:44:00.945000 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:00.945006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:00.945108 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:00.945193 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:00.945991 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.947111 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.949217 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.949635 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.949743 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:00.950174 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:00.950273 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:00.950349 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:00.950515 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 16:44:00.950610 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260812 16:44:00.950982 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:00.942440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.31.61.190:42667)
0812 16:44:00.944444 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:00.944450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:00.944472 (+ 22us) server_negotiation.cc:407] Connection header received
0812 16:44:00.944610 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:00.944616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:00.944842 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:00.945017 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:00.945506 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.946396 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.948257 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.948769 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.948890 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:00.949487 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:00.949952 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:00.950069 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:00.950598 (+ 529us) server_negotiation.cc:299] Negotiation successful
0812 16:44:00.950688 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1881,"thread_start_us":1776,"threads_started":1}
I20260812 16:44:00.953965 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:00.943269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.31.61.188:37423)
0812 16:44:00.943441 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:00.943447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:00.948215 (+ 4768us) server_negotiation.cc:407] Connection header received
0812 16:44:00.948472 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:00.948477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:00.948565 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:00.948651 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:00.949305 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.950254 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.952064 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:00.952665 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:00.952796 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:00.953271 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:00.953407 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:00.953480 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:00.953673 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 16:44:00.953745 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 16:44:00.960848 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22298
I20260812 16:44:00.961995 22361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:01.070376 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 22298
W20260812 16:44:01.089056 22469 connection.cc:535] server connection from 127.0.0.1:47834 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 16:44:01.089124 22391 connection.cc:535] server connection from 127.0.0.1:38720 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 16:44:01.089700 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22373
I20260812 16:44:01.090832 22436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:01.202407 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 22373
I20260812 16:44:01.220993 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22449
I20260812 16:44:01.222361 22513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:01.348539 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 22449
2026-08-12T16:44:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5358 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 16:44:01 dist-test-slave-f7th krb5kdc[22683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 16:44:01 dist-test-slave-f7th krb5kdc[22683](info): set up 2 sockets
Aug 12 16:44:01 dist-test-slave-f7th krb5kdc[22683](info): commencing operation
krb5kdc: starting...
W20260812 16:44:02.395849 31990 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/test-user.keytab.
Aug 12 16:44:02 dist-test-slave-f7th krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T16:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T16:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.61.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
Entry for principal kudu/127.31.61.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab.
I20260812 16:44:02.673179 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36085
--webserver_interface=127.31.61.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36085,127.31.61.189:40599,127.31.61.188:42817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 16:44:03.010797 22696 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:44:03.011163 22696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:03.011260 22696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:03.011377 22696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 16:44:03.018864 22696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:03.018971 22696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:03.019055 22696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:03.019132 22696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:03.019217 22696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:03.030349 22696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36085,127.31.61.189:40599,127.31.61.188:42817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.190.keytab
--principal=kudu/127.31.61.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.190:36085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.61.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:03.031734 22696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:03.033061 22696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:03.044525 22701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:03.044440 22702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:03.044489 22704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:03.045717 22696 server_base.cc:1034] running on GCE node
I20260812 16:44:03.046525 22696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:03.047968 22696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:03.049203 22696 hybrid_clock.cc:648] HybridClock initialized: now 1786553043049131 us; error 46 us; skew 500 ppm
I20260812 16:44:03.049539 22696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:03 dist-test-slave-f7th krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:03.052642 22696 init.cc:377] Logged in from keytab as kudu/127.31.61.190@KRBTEST.COM (short username kudu)
I20260812 16:44:03.054217 22696 webserver.cc:460] Webserver started at http://127.31.61.190:34387/ using document root <none> and password file <none>
I20260812 16:44:03.054899 22696 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:03.055002 22696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:03.055332 22696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:03.058388 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/data/instance:
uuid: "0bdd3d2d910a45a0a6f7796e87e63afb"
format_stamp: "Formatted at 2026-08-12 16:44:03 on dist-test-slave-f7th"
server_key: "8a9426d1df20e309ef38b64a3c74d0ce"
server_key_iv: "7d9e6d022f5ac652522682c8475845e3"
server_key_version: "encryptionkey@0"
I20260812 16:44:03.059163 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/wal/instance:
uuid: "0bdd3d2d910a45a0a6f7796e87e63afb"
format_stamp: "Formatted at 2026-08-12 16:44:03 on dist-test-slave-f7th"
server_key: "8a9426d1df20e309ef38b64a3c74d0ce"
server_key_iv: "7d9e6d022f5ac652522682c8475845e3"
server_key_version: "encryptionkey@0"
I20260812 16:44:03.064294 22696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:44:03.067624 22711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:03.068872 22696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 16:44:03.069044 22696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/wal
uuid: "0bdd3d2d910a45a0a6f7796e87e63afb"
format_stamp: "Formatted at 2026-08-12 16:44:03 on dist-test-slave-f7th"
server_key: "8a9426d1df20e309ef38b64a3c74d0ce"
server_key_iv: "7d9e6d022f5ac652522682c8475845e3"
server_key_version: "encryptionkey@0"
I20260812 16:44:03.069233 22696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:03.094368 22696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:03.098482 22696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:03.098744 22696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:03.116816 22696 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.190:36085
I20260812 16:44:03.116839 22765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.190:36085 every 8 connection(s)
I20260812 16:44:03.118160 22696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/data/info.pb
I20260812 16:44:03.120628 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 22696
I20260812 16:44:03.120797 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-0/wal/instance
I20260812 16:44:03.121248 31990 external_mini_cluster.cc:1428] Setting key a0be0cfbf50ac923c5129c60165efae4
I20260812 16:44:03.122037 22766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:03.128849 22766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:44:03 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.61.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.189@KRBTEST.COM" created.
I20260812 16:44:03.152554 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.133773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.31.61.190:36085)
0812 16:44:03.134462 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:03.134471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:03.134531 (+ 60us) server_negotiation.cc:407] Connection header received
0812 16:44:03.135770 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:03.135775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:03.136018 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:03.136186 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:03.137488 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.138693 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.140395 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.140963 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.141173 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:03.144820 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:03.144829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:03.144842 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:03.144874 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:03.147964 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:03.148734 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:03.148743 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:03.148746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:03.148940 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:03.149359 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:03.149364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:03.149367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:03.150070 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:03.150270 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:03.150969 (+ 699us) server_negotiation.cc:299] Negotiation successful
0812 16:44:03.151435 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":473,"thread_start_us":323,"threads_started":1}
I20260812 16:44:03.152560 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.132474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36085 (local address 127.0.0.1:43256)
0812 16:44:03.133293 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.133336 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:03.135395 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:03.136228 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:03.136234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:03.136930 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:03.137274 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:03.137291 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.138843 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.138847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:03.140238 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:03.140252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.141203 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.141210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:03.141416 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:03.142017 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:03.142041 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:03.144598 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:03.148114 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:03.148126 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:03.148136 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:03.148576 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:03.149078 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:03.149082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:03.149084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:03.149250 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:03.150266 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:03.150273 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:03.150548 (+ 275us) client_negotiation.cc:769] Sending connection context
0812 16:44:03.150798 (+ 250us) client_negotiation.cc:240] Negotiation successful
0812 16:44:03.151260 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":321,"threads_started":1}
I20260812 16:44:03.155984 22766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:44:03.157157 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.156626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34368)
0812 16:44:03.156809 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.156940 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 16:44:03.157454 22716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
I20260812 16:44:03.159044 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.158656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34380)
0812 16:44:03.158812 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.158891 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 16:44:03.159258 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:40599: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab.
Entry for principal kudu/127.31.61.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab.
I20260812 16:44:03.183149 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.189:40599
--webserver_interface=127.31.61.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab
--principal=kudu/127.31.61.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36085,127.31.61.189:40599,127.31.61.188:42817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:44:03.208638 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } attempt: 1
I20260812 16:44:03.210024 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.209467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34394)
0812 16:44:03.209706 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.209837 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 16:44:03.211522 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.211173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34398)
0812 16:44:03.211307 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.211384 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 16:44:03.211719 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:40599: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
I20260812 16:44:03.279976 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } attempt: 2
I20260812 16:44:03.281479 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.280797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34412)
0812 16:44:03.281205 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.281311 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260812 16:44:03.282640 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.282246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34424)
0812 16:44:03.282417 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.282510 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 16:44:03.282860 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:40599: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
I20260812 16:44:03.374158 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } attempt: 3
I20260812 16:44:03.375357 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.374894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34434)
0812 16:44:03.375124 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.375220 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 16:44:03.376632 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.376188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34440)
0812 16:44:03.376332 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.376466 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 16:44:03.376858 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:40599: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
W20260812 16:44:03.514948 22773 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:44:03.515375 22773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:03.515470 22773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:03.515575 22773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 16:44:03.520113 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } attempt: 4
I20260812 16:44:03.521384 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.520823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34450)
0812 16:44:03.521158 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.521249 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260812 16:44:03.522315 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.521998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34454)
0812 16:44:03.522115 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.522197 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 16:44:03.522500 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.189:40599: Network error: Client connection negotiation failed: client connection to 127.31.61.189:40599: connect: Connection refused (error 111)
W20260812 16:44:03.523548 22773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:03.523665 22773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:03.523753 22773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:03.523829 22773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:03.523912 22773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:03.534114 22773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36085,127.31.61.189:40599,127.31.61.188:42817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.189.keytab
--principal=kudu/127.31.61.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.189:40599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.61.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:03.535427 22773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:03.536759 22773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:03.548436 22782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:03.548395 22779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:03.548470 22780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:03.549642 22773 server_base.cc:1034] running on GCE node
I20260812 16:44:03.550411 22773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:03.551848 22773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:03.553097 22773 hybrid_clock.cc:648] HybridClock initialized: now 1786553043553019 us; error 45 us; skew 500 ppm
I20260812 16:44:03.553427 22773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:03 dist-test-slave-f7th krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:03.556568 22773 init.cc:377] Logged in from keytab as kudu/127.31.61.189@KRBTEST.COM (short username kudu)
I20260812 16:44:03.558053 22773 webserver.cc:460] Webserver started at http://127.31.61.189:45645/ using document root <none> and password file <none>
I20260812 16:44:03.558763 22773 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:03.558867 22773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:03.559173 22773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:03.562165 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/data/instance:
uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9"
format_stamp: "Formatted at 2026-08-12 16:44:03 on dist-test-slave-f7th"
server_key: "1b5ca2162febc09b8e4d1feee5dd5bf9"
server_key_iv: "f8eb7bc8ebc804274fafaa5ea283ec17"
server_key_version: "encryptionkey@0"
I20260812 16:44:03.562963 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/wal/instance:
uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9"
format_stamp: "Formatted at 2026-08-12 16:44:03 on dist-test-slave-f7th"
server_key: "1b5ca2162febc09b8e4d1feee5dd5bf9"
server_key_iv: "f8eb7bc8ebc804274fafaa5ea283ec17"
server_key_version: "encryptionkey@0"
I20260812 16:44:03.568251 22773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 16:44:03.571763 22789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:03.573194 22773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 16:44:03.573365 22773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/wal
uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9"
format_stamp: "Formatted at 2026-08-12 16:44:03 on dist-test-slave-f7th"
server_key: "1b5ca2162febc09b8e4d1feee5dd5bf9"
server_key_iv: "f8eb7bc8ebc804274fafaa5ea283ec17"
server_key_version: "encryptionkey@0"
I20260812 16:44:03.573616 22773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:03.588153 22773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:03.592319 22773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:03.592550 22773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:03.610285 22773 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.189:40599
I20260812 16:44:03.610348 22841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.189:40599 every 8 connection(s)
I20260812 16:44:03.611717 22773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/data/info.pb
I20260812 16:44:03.615433 22842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:03.620776 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 22773
I20260812 16:44:03.620960 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-1/wal/instance
I20260812 16:44:03.621496 31990 external_mini_cluster.cc:1428] Setting key 3176883c05c1eab1a46735c4cff771d3
I20260812 16:44:03.622542 22842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:44:03 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:44:03.643872 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.625781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.31.61.190:36085)
0812 16:44:03.625986 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:03.625992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:03.627266 (+ 1274us) server_negotiation.cc:407] Connection header received
0812 16:44:03.629678 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:03.629683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:03.629757 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:03.629846 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:03.631738 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.632418 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.633913 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.634357 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.634472 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:03.637892 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:03.637897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:03.637901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:03.637936 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:03.640601 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:03.641361 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:03.641366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:03.641369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:03.641492 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:03.641917 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:03.641921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:03.641923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:03.642226 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:03.642499 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:03.643252 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 16:44:03.643500 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:44:03.644881 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.626260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36085 (local address 127.0.0.1:43272)
0812 16:44:03.627072 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.627123 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:03.629363 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:03.630095 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:03.630107 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:03.630814 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:03.631539 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:03.631561 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.632588 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.632593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:03.633753 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:03.633765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.634597 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.634605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:03.634817 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:03.635467 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:03.635515 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:03.637663 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:03.640744 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:03.640754 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:03.640773 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:03.641230 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:03.641602 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:03.641607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:03.641611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:03.641801 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:03.642444 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:03.642453 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:03.642863 (+ 410us) client_negotiation.cc:769] Sending connection context
0812 16:44:03.643116 (+ 253us) client_negotiation.cc:240] Negotiation successful
0812 16:44:03.643713 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":333,"threads_started":1}
I20260812 16:44:03.646780 22842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.61.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.61.188@KRBTEST.COM" created.
Aug 12 16:44:03 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:44:03.661029 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.647440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34460)
0812 16:44:03.647680 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.647696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:03.647779 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:03.648763 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:03.648769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:03.649151 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:03.649290 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:03.649299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.650697 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.650702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:03.651807 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:03.651820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.652899 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.652903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:03.653051 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:03.653594 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:03.653611 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:03.655554 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:03.658607 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:03.658613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:03.658617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:03.658967 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:03.659333 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:03.659338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:03.659342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:03.659462 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:03.660085 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:03.660090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:03.660245 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 16:44:03.660355 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 16:44:03.660632 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 16:44:03.661134 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.647745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.31.61.189:40599)
0812 16:44:03.648306 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:03.648312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:03.648337 (+ 25us) server_negotiation.cc:407] Connection header received
0812 16:44:03.648377 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:03.648383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:03.648575 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:03.648688 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:03.649461 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.650559 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.651940 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.652760 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.652894 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:03.655767 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:03.655779 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:03.655793 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:03.655827 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:03.658481 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:03.659105 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:03.659112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:03.659114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:03.659231 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:03.659562 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:03.659566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:03.659568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:03.659952 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:03.660150 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:03.660543 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 16:44:03.660770 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":282,"threads_started":1}
I20260812 16:44:03.663659 22842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:44:03.664835 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.664381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40094)
0812 16:44:03.664568 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.664679 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 16:44:03.665126 22792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
I20260812 16:44:03.666610 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.666277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40106)
0812 16:44:03.666421 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.666503 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 16:44:03.666818 22842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:42817: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.61.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab.
Entry for principal kudu/127.31.61.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab.
I20260812 16:44:03.683838 31990 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
/tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.188:42817
--webserver_interface=127.31.61.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab
--principal=kudu/127.31.61.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.61.148:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.61.190:36085,127.31.61.189:40599,127.31.61.188:42817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 16:44:03.716145 22842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } attempt: 1
I20260812 16:44:03.717338 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.716865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40112)
0812 16:44:03.717072 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.717170 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 16:44:03.718650 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.718178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40118)
0812 16:44:03.718333 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.718434 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 16:44:03.718873 22842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:42817: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
I20260812 16:44:03.787114 22842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } attempt: 2
I20260812 16:44:03.788437 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.787919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40132)
0812 16:44:03.788184 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.788281 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 16:44:03.789425 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.789111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40136)
0812 16:44:03.789242 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.789313 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 16:44:03.789640 22842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:42817: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
I20260812 16:44:03.821748 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } attempt: 5
Aug 12 16:44:03 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:44:03.835707 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.822698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.31.61.189:40599)
0812 16:44:03.822900 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:03.822905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:03.822940 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:44:03.823005 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:03.823011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:03.823110 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:03.823217 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:03.824052 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.825179 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.826367 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.827073 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.827200 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:03.830398 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:03.830403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:03.830407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:03.830441 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:03.832903 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:03.833684 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:03.833688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:03.833691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:03.833818 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:03.834342 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:03.834346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:03.834348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:03.834645 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:03.834761 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:03.835248 (+ 487us) server_negotiation.cc:299] Negotiation successful
0812 16:44:03.835424 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 16:44:03.835958 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.822546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34462)
0812 16:44:03.822765 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.822787 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:03.822904 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:03.823236 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:03.823239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:03.823695 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:03.823888 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:03.823899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.825360 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.825366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:03.826217 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:03.826229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:03.827205 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:03.827210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:03.827362 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:03.827956 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:03.827974 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:03.830205 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:03.833036 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:03.833042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:03.833046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:03.833557 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:03.833978 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:03.833985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:03.833989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:03.834203 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:03.834782 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:03.834788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:03.834988 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 16:44:03.835176 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 16:44:03.835498 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 16:44:03.837122 22766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:44:03.838321 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.837930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40142)
0812 16:44:03.838103 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.838182 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 16:44:03.839491 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.839128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40158)
0812 16:44:03.839291 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.839380 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 16:44:03.839725 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:42817: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
I20260812 16:44:03.880860 22842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } attempt: 3
I20260812 16:44:03.882272 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.881605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40170)
0812 16:44:03.881891 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.882015 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260812 16:44:03.883409 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.882974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40176)
0812 16:44:03.883207 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.883291 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260812 16:44:03.883633 22842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:42817: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
I20260812 16:44:03.890975 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } attempt: 1
I20260812 16:44:03.892227 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.891748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40178)
0812 16:44:03.891959 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.892051 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 16:44:03.893188 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.892852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40192)
0812 16:44:03.893000 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.893077 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 16:44:03.893380 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:42817: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
I20260812 16:44:03.961606 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } attempt: 2
I20260812 16:44:03.962776 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.962250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40200)
0812 16:44:03.962486 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.962580 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 16:44:03.963922 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:03.963536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40216)
0812 16:44:03.963709 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:03.963798 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 16:44:03.964141 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:42817: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
W20260812 16:44:04.021191 22849 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 16:44:04.021564 22849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 16:44:04.021677 22849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 16:44:04.021793 22849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 16:44:04.026850 22842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } attempt: 4
I20260812 16:44:04.028023 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.027476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40226)
0812 16:44:04.027756 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:04.027843 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 16:44:04.029016 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.028699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40230)
0812 16:44:04.028841 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:04.028914 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 16:44:04.029206 22842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:42817: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
W20260812 16:44:04.029418 22849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 16:44:04.029510 22849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 16:44:04.029609 22849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 16:44:04.029680 22849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 16:44:04.029767 22849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 16:44:04.039938 22849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.61.148:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.61.190:36085,127.31.61.189:40599,127.31.61.188:42817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/krb5kdc/kudu_127.31.61.188.keytab
--principal=kudu/127.31.61.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.61.188:42817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.61.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 15:17:36 UTC on ccycloud.quasar-sxjchh.root.comops.site
build id 4582766
ASAN enabled
I20260812 16:44:04.041301 22849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 16:44:04.042654 22849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 16:44:04.054698 22855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:04.054864 22857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 16:44:04.055245 22854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 16:44:04.055675 22849 server_base.cc:1034] running on GCE node
I20260812 16:44:04.056507 22849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 16:44:04.057960 22849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 16:44:04.059185 22849 hybrid_clock.cc:648] HybridClock initialized: now 1786553044059118 us; error 37 us; skew 500 ppm
I20260812 16:44:04.059526 22849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 16:44:04 dist-test-slave-f7th krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553044, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 16:44:04.062568 22849 init.cc:377] Logged in from keytab as kudu/127.31.61.188@KRBTEST.COM (short username kudu)
I20260812 16:44:04.064019 22849 webserver.cc:460] Webserver started at http://127.31.61.188:42555/ using document root <none> and password file <none>
I20260812 16:44:04.064715 22849 fs_manager.cc:359] Metadata directory not provided
I20260812 16:44:04.064808 22849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 16:44:04.065120 22849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 16:44:04.068207 22849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/data/instance:
uuid: "7852c1a2838e4032b596c131e4ffe580"
format_stamp: "Formatted at 2026-08-12 16:44:04 on dist-test-slave-f7th"
server_key: "90dcb5086b3eeec137117e6320648ce3"
server_key_iv: "131e8104a45ba5cc7f78d51978957576"
server_key_version: "encryptionkey@0"
I20260812 16:44:04.069031 22849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/wal/instance:
uuid: "7852c1a2838e4032b596c131e4ffe580"
format_stamp: "Formatted at 2026-08-12 16:44:04 on dist-test-slave-f7th"
server_key: "90dcb5086b3eeec137117e6320648ce3"
server_key_iv: "131e8104a45ba5cc7f78d51978957576"
server_key_version: "encryptionkey@0"
I20260812 16:44:04.070331 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } attempt: 3
I20260812 16:44:04.071472 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.070899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40238)
0812 16:44:04.071177 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:04.071283 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 16:44:04.072603 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.072166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40252)
0812 16:44:04.072379 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:04.072468 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260812 16:44:04.072892 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.61.188:42817: Network error: Client connection negotiation failed: client connection to 127.31.61.188:42817: connect: Connection refused (error 111)
I20260812 16:44:04.074925 22849 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 16:44:04.078408 22864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 16:44:04.079838 22849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 16:44:04.079990 22849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/data,/tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/wal
uuid: "7852c1a2838e4032b596c131e4ffe580"
format_stamp: "Formatted at 2026-08-12 16:44:04 on dist-test-slave-f7th"
server_key: "90dcb5086b3eeec137117e6320648ce3"
server_key_iv: "131e8104a45ba5cc7f78d51978957576"
server_key_version: "encryptionkey@0"
I20260812 16:44:04.080201 22849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 16:44:04.094038 22849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 16:44:04.098652 22849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 16:44:04.098965 22849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 16:44:04.118326 22849 rpc_server.cc:307] RPC server started. Bound to: 127.31.61.188:42817
I20260812 16:44:04.118358 22917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.61.188:42817 every 8 connection(s)
I20260812 16:44:04.119805 22849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/data/info.pb
I20260812 16:44:04.124531 22918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 16:44:04.130125 31990 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu as pid 22849
I20260812 16:44:04.130288 31990 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgT9LAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786552676633950-31990-0/minicluster-data/master-2/wal/instance
I20260812 16:44:04.130723 31990 external_mini_cluster.cc:1428] Setting key baf69f224114c4eb1d3b54490a4ea6c9
I20260812 16:44:04.132036 22918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:44:04 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553044, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
Aug 12 16:44:04 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
I20260812 16:44:04.154877 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.135118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.31.61.190:36085)
0812 16:44:04.135392 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.135399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.136081 (+ 682us) server_negotiation.cc:407] Connection header received
0812 16:44:04.136317 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.136323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.136406 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.136508 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.138424 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.139649 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.141926 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.142677 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.142830 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.147339 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.147348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.147353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.147402 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.150648 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.151557 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.151564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.151568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.151784 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.152343 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.152348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.152351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.152742 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.152851 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.153880 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.154213 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 16:44:04.155718 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.132652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.31.61.188:42817)
0812 16:44:04.133444 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.133459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.133511 (+ 52us) server_negotiation.cc:407] Connection header received
0812 16:44:04.135588 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.135654 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.136042 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.136405 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.138340 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.140103 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.142048 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.143037 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.143146 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.147907 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.147916 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.147930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.147987 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.151006 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.151964 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.151973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.151976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.152184 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.152684 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.152688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.152690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.153427 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.153735 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.154132 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.154355 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":555,"thread_start_us":326,"threads_started":1}
I20260812 16:44:04.155754 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.134939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.190:36085 (local address 127.0.0.1:43276)
0812 16:44:04.135920 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:04.135969 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:04.136132 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:04.136634 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:04.136641 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:04.137627 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:04.138183 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.138210 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.139841 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.139845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.141747 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.141765 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.142941 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.142947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.143149 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.144143 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:04.144175 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:04.147087 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:04.150837 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.150850 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.150865 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.151371 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.151958 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.151963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.151967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.152224 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.152928 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:04.152937 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:04.153427 (+ 490us) client_negotiation.cc:769] Sending connection context
0812 16:44:04.153723 (+ 296us) client_negotiation.cc:240] Negotiation successful
0812 16:44:04.154231 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":57,"spinlock_wait_cycles":42496,"thread_start_us":572,"threads_started":1}
I20260812 16:44:04.158325 22918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 16:44:04 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553044, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
Aug 12 16:44:04 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.190@KRBTEST.COM
I20260812 16:44:04.172968 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.159101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.31.61.189:40599)
0812 16:44:04.159276 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.159281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.159300 (+ 19us) server_negotiation.cc:407] Connection header received
0812 16:44:04.159455 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.159460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.159530 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.159656 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.160316 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.161250 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.162643 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.163323 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.163427 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.166812 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.166817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.166821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.166856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.170114 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.170909 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.170915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.170919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.171089 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.171525 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.171530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.171533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.171929 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.172098 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.172490 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.172684 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:44:04.172988 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.158982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.189:40599 (local address 127.0.0.1:34474)
0812 16:44:04.159219 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:04.159236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:04.159336 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:04.159688 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:04.159693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:04.160034 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:04.160173 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.160181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.161387 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.161393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.162476 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.162487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.163472 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.163478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.163728 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.164409 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:04.164428 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:04.166610 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:04.170268 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.170271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.170274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.170770 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.171225 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.171229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.171231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.171387 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.172066 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:04.172070 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:04.172209 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 16:44:04.172344 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 16:44:04.172612 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260812 16:44:04.174218 22918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } has no permanent_uuid. Determining permanent_uuid...
I20260812 16:44:04.175626 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.159084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.31.61.190:36085)
0812 16:44:04.159255 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.159261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.159278 (+ 17us) server_negotiation.cc:407] Connection header received
0812 16:44:04.159432 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.159437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.159503 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.159580 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.160941 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.161897 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.163339 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.163994 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.164102 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.168980 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.168988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.168993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.169037 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.172415 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.173274 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.173279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.173281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.173455 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.173918 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.173924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.173928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.174406 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.174716 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.175036 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.175287 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 12 16:44:04 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553044, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.188@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
Aug 12 16:44:04 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.61.189@KRBTEST.COM
I20260812 16:44:04.188704 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.175243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40264)
0812 16:44:04.175760 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:04.175779 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:04.175886 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:04.176167 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:04.176171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:04.176550 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:04.176717 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.176729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.177858 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.177864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.178789 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.178802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.179442 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.179448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.179674 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.180253 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:04.180271 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:04.182415 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:04.186043 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.186048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.186052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.186617 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.186979 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.186984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.186987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.187161 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.187708 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:04.187713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:04.187847 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 16:44:04.187985 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 16:44:04.188269 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422}
I20260812 16:44:04.188711 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.175298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.31.61.188:42817)
0812 16:44:04.175544 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.175550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.175832 (+ 282us) server_negotiation.cc:407] Connection header received
0812 16:44:04.175982 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.175987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.176048 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.176124 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.176829 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.177765 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.178913 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.179308 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.179417 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.182647 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.182655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.182659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.182699 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.185827 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.186733 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.186738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.186740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.186866 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.187273 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.187276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.187278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.187554 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.187739 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.188088 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.188427 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":135}
I20260812 16:44:04.193022 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.177406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.31.61.189:40599)
0812 16:44:04.177566 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.177572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.177663 (+ 91us) server_negotiation.cc:407] Connection header received
0812 16:44:04.177801 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.177805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.177874 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.177973 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.179371 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.180293 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.181595 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.182199 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.182362 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.187637 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.187645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.187649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.187684 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.190559 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.191177 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.191183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.191186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.191321 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.191693 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.191698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.191701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.191953 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.192228 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.192504 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.192736 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 16:44:04.193919 22918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580: Bootstrap starting.
I20260812 16:44:04.197350 22918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:04.197494 31990 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 16:44:04.198472 22918 log.cc:824] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:04.201735 22918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580: No bootstrap required, opened a new log
I20260812 16:44:04.200027 22741 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43282
I20260812 16:44:04.212656 22918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } }
I20260812 16:44:04.213140 22918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7852c1a2838e4032b596c131e4ffe580, State: Initialized, Role: FOLLOWER
I20260812 16:44:04.213685 22918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } }
I20260812 16:44:04.214749 22924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.215063 22924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:44:04.215960 22918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 16:44:04.226127 22936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:44:04.226316 22936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:44:04.222926 22849 master_runner.cc:186] Error getting master registration for 127.31.61.190:36085: OK, instance_id { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" instance_seqno: 1786553043111743 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:44:04.230598 22849 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 16:44:04.250149 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } attempt: 4
Aug 12 16:44:04 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.190@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
I20260812 16:44:04.265156 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.250903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40268)
0812 16:44:04.251117 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:04.251140 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:04.251288 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:04.251672 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:04.251678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:04.252201 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:04.252415 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.252428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.253869 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.253875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.255345 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.255362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.256424 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.256430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.256648 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.257280 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:04.257298 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:04.259551 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:04.262602 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.262608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.262613 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.263003 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.263358 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.263363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.263367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.263538 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.264171 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:04.264175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:04.264336 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 16:44:04.264461 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 16:44:04.264763 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 16:44:04.265159 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.250960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.31.61.188:42817)
0812 16:44:04.251147 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.251153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.251248 (+ 95us) server_negotiation.cc:407] Connection header received
0812 16:44:04.251404 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.251410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.251507 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.251651 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.252583 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.253716 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.255520 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.256274 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.256405 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.259759 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.259764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.259767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.259801 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.262456 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.263118 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.263122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.263125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.263246 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.263674 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.263678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.263680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.264049 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.264190 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.264587 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.264797 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 16:44:04.269936 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb: Bootstrap starting.
I20260812 16:44:04.273384 22766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:04.274720 22766 log.cc:824] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:04.277894 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb: No bootstrap required, opened a new log
I20260812 16:44:04.283123 22766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } }
I20260812 16:44:04.283504 22766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdd3d2d910a45a0a6f7796e87e63afb, State: Initialized, Role: FOLLOWER
I20260812 16:44:04.283982 22766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } }
I20260812 16:44:04.285094 22939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.285486 22939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:44:04.286329 22766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:04.286448 22741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 16:44:04.286682 22741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:04.290596 22741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } }
I20260812 16:44:04.293483 22893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7852c1a2838e4032b596c131e4ffe580"
I20260812 16:44:04.294035 22893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:04.297773 22893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bdd3d2d910a45a0a6f7796e87e63afb in term 1.
I20260812 16:44:04.298650 22714 leader_election.cc:302] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bdd3d2d910a45a0a6f7796e87e63afb, 7852c1a2838e4032b596c131e4ffe580; no voters:
I20260812 16:44:04.299226 22939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 16:44:04.310699 22741 leader_election.cc:288] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [CANDIDATE]: Term 1 election: Requested vote from peers dde2aa0ac99b47c69b8ada7ce7ab53c9 (127.31.61.189:40599), 7852c1a2838e4032b596c131e4ffe580 (127.31.61.188:42817)
I20260812 16:44:04.316252 22939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [term 1 LEADER]: Becoming Leader. State: Replica: 0bdd3d2d910a45a0a6f7796e87e63afb, State: Running, Role: LEADER
I20260812 16:44:04.317175 22939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } }
I20260812 16:44:04.328392 22842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } attempt: 5
I20260812 16:44:04.345820 22952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bdd3d2d910a45a0a6f7796e87e63afb. Latest consensus state: current_term: 1 leader_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.346289 22952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:04.326844 22696 master_runner.cc:186] Error getting master registration for 127.31.61.189:40599: OK, instance_id { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" instance_seqno: 1786553043605242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 16:44:04.348829 22696 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 16:44:04.351420 22871 tablet.cc:2369] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:44:04.351778 22718 tablet.cc:2369] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 16:44:04.366335 22951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 16:44:04.366487 22951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 16:44:04.366629 22962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 12 16:44:04 dist-test-slave-f7th krb5kdc[22683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786553043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.61.189@KRBTEST.COM for kudu/127.31.61.188@KRBTEST.COM
I20260812 16:44:04.368845 22962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 16:44:04.388010 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.328931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.61.188:42817 (local address 127.0.0.1:40290)
0812 16:44:04.329193 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 16:44:04.329213 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 16:44:04.329324 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 16:44:04.353373 (+ 24049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 16:44:04.353380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 16:44:04.353982 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0812 16:44:04.354241 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.354258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.359918 (+ 5660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.359926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.361503 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 16:44:04.361528 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.363532 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.363541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 16:44:04.363837 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.364623 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 16:44:04.364652 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 16:44:04.369793 (+ 5141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 16:44:04.383477 (+ 13684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.383485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.383489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.384182 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.385496 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 16:44:04.385501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 16:44:04.385505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 16:44:04.385695 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 16:44:04.386864 (+ 1169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 16:44:04.386870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 16:44:04.387047 (+ 177us) client_negotiation.cc:769] Sending connection context
0812 16:44:04.387232 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 16:44:04.387564 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260812 16:44:04.394611 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.329269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.31.61.188:42817)
0812 16:44:04.352747 (+ 23478us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.352755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.352790 (+ 35us) server_negotiation.cc:407] Connection header received
0812 16:44:04.352867 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.352872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.352990 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.354426 (+ 1436us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.354752 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.359691 (+ 4939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.361751 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.362549 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.364254 (+ 1705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.370101 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.370108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.370114 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.370164 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.376341 (+ 6177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.384452 (+ 8111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.384459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.384463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.384660 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.385847 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.385852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.385855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.386289 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.388830 (+ 2541us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.391924 (+ 3094us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.392208 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23309,"thread_start_us":23861,"threads_started":1}
I20260812 16:44:04.396106 22893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [term 1 FOLLOWER]: Refusing update from remote peer 0bdd3d2d910a45a0a6f7796e87e63afb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:04.397289 22952 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:04.401921 22842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9: Bootstrap starting.
I20260812 16:44:04.408963 22842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9: Neither blocks nor log segments found. Creating new log.
I20260812 16:44:04.410842 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.329046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.31.61.188:42817)
0812 16:44:04.329667 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.329674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.356305 (+ 26631us) server_negotiation.cc:407] Connection header received
0812 16:44:04.356600 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.356606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.356789 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.356880 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.366586 (+ 9706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.368795 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.376552 (+ 7757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.377344 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.377451 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.392660 (+ 15209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.392667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.392672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.392709 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.397817 (+ 5108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.407846 (+ 10029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.407853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.407856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.408045 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.408536 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.408542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.408546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.408949 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.409206 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.410222 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.410510 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492}
I20260812 16:44:04.423820 22842 log.cc:824] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9: Log is configured to *not* fsync() on all Append() calls
I20260812 16:44:04.424997 22924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bdd3d2d910a45a0a6f7796e87e63afb. Latest consensus state: current_term: 1 leader_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.425338 22924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:44:04.426734 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.378546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.31.61.189:40599)
0812 16:44:04.378735 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.378741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.385614 (+ 6873us) server_negotiation.cc:407] Connection header received
0812 16:44:04.386014 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.386020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.386096 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.386160 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.387949 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.388999 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.390743 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.391382 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.391479 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.412642 (+ 21163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.412650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.412656 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.412705 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.416351 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.420397 (+ 4046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.420405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.420409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.420600 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.422469 (+ 1869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.422474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.422478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.422861 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.423112 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.426146 (+ 3034us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.426408 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 16:44:04.436283 22952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.436715 22952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:04.439845 22962 catalog_manager.cc:1347] Generated new cluster ID: 04d053e390d349e48d2755613d9d67de
I20260812 16:44:04.439952 22962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 16:44:04.439994 22842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9: No bootstrap required, opened a new log
I20260812 16:44:04.447643 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.328522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.31.61.190:36085)
0812 16:44:04.328732 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.328739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.378391 (+ 49652us) server_negotiation.cc:407] Connection header received
0812 16:44:04.382027 (+ 3636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.382035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.382159 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.382235 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.389145 (+ 6910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.390333 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.393269 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.394007 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.394117 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.430070 (+ 35953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 16:44:04.430079 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 16:44:04.430085 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 16:44:04.430153 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 16:44:04.434043 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.434987 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.434997 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.435001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.435213 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 16:44:04.435706 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 16:44:04.435713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 16:44:04.435717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 16:44:04.437100 (+ 1383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 16:44:04.437378 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.438304 (+ 926us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.438789 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 16:44:04.451445 22842 raft_consensus.cc:348] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } }
I20260812 16:44:04.454515 22842 raft_consensus.cc:729] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde2aa0ac99b47c69b8ada7ce7ab53c9, State: Initialized, Role: FOLLOWER
I20260812 16:44:04.455101 22842 consensus_queue.cc:260] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } }
I20260812 16:44:04.456847 22924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.456432 22970 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.457095 22924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:44:04.457172 22970 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:44:04.457578 22842 sys_catalog.cc:564] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 16:44:04.457967 22816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 16:44:04.458804 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9"
I20260812 16:44:04.463353 22816 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [term 1 FOLLOWER]: Refusing update from remote peer 0bdd3d2d910a45a0a6f7796e87e63afb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 16:44:04.470883 22817 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bdd3d2d910a45a0a6f7796e87e63afb in term 1.
I20260812 16:44:04.472359 22962 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 16:44:04.473464 22939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 16:44:04.474220 22962 catalog_manager.cc:1504] Loading token signing keys...
I20260812 16:44:04.493841 22962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb: Generated new TSK 0
I20260812 16:44:04.494915 22962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 16:44:04.540786 22970 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bdd3d2d910a45a0a6f7796e87e63afb. Latest consensus state: current_term: 1 leader_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.541222 22970 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:44:04.541687 22979 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.542075 22979 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 16:44:04.547353 22952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd3d2d910a45a0a6f7796e87e63afb" member_type: VOTER last_known_addr { host: "127.31.61.190" port: 36085 } } peers { permanent_uuid: "dde2aa0ac99b47c69b8ada7ce7ab53c9" member_type: VOTER last_known_addr { host: "127.31.61.189" port: 40599 } } peers { permanent_uuid: "7852c1a2838e4032b596c131e4ffe580" member_type: VOTER last_known_addr { host: "127.31.61.188" port: 42817 } } }
I20260812 16:44:04.547776 22952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdd3d2d910a45a0a6f7796e87e63afb [sys.catalog]: This master's current role is: LEADER
I20260812 16:44:04.576437 22987 catalog_manager.cc:1259] Loaded cluster ID: 04d053e390d349e48d2755613d9d67de
I20260812 16:44:04.576615 22987 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9: loading cluster ID for follower catalog manager: success
I20260812 16:44:04.580468 22987 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9: acquiring CA information for follower catalog manager: success
I20260812 16:44:04.586557 22987 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dde2aa0ac99b47c69b8ada7ce7ab53c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 16:44:04.665184 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:04.653385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.31.61.188:42817)
0812 16:44:04.653583 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.653589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.661636 (+ 8047us) server_negotiation.cc:407] Connection header received
0812 16:44:04.663146 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.663153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.663250 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.663324 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.664824 (+ 1500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40304: BlockingRecv error: recv got EOF from 127.0.0.1:40304 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 16:44:04.668866 22770 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:04.652965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.31.61.190:36085)
0812 16:44:04.653199 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.653207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.666696 (+ 13489us) server_negotiation.cc:407] Connection header received
0812 16:44:04.667402 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.667409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.667530 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.667632 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.668614 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43310: BlockingRecv error: recv got EOF from 127.0.0.1:43310 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 16:44:04.673278 22846 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:04.662294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.31.61.189:40599)
0812 16:44:04.662469 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.662474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.670196 (+ 7722us) server_negotiation.cc:407] Connection header received
0812 16:44:04.670869 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.670875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.670987 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.671071 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.673007 (+ 1936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34506: BlockingRecv error: recv got EOF from 127.0.0.1:34506 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 16:44:04.723212 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.707258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.31.61.189:40599)
0812 16:44:04.707450 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.707456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.714361 (+ 6905us) server_negotiation.cc:407] Connection header received
0812 16:44:04.714928 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.714935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.715188 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.715347 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:04.716102 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.717489 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.719561 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.720259 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.720357 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.721694 (+ 1337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:04.722394 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:04.722514 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.722755 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.722867 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 16:44:04.766736 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:04.756675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.31.61.188:42817)
0812 16:44:04.756902 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.756908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.764495 (+ 7587us) server_negotiation.cc:407] Connection header received
0812 16:44:04.765093 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.765100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.765250 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.765332 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.766501 (+ 1169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40314: BlockingRecv error: recv got EOF from 127.0.0.1:40314 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:44:04.815395 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.801120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.31.61.189:40599)
0812 16:44:04.801369 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.801375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.808258 (+ 6883us) server_negotiation.cc:407] Connection header received
0812 16:44:04.808781 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.808791 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.808917 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.808983 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:04.809724 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.810902 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.812986 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.813727 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.813825 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.814514 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:04.814675 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:04.814737 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.814981 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.815077 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 16:44:04.861976 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:04.852332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.31.61.188:42817)
0812 16:44:04.852712 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.852720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.860331 (+ 7611us) server_negotiation.cc:407] Connection header received
0812 16:44:04.860551 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.860557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.860673 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.860796 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.861737 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40330: BlockingRecv error: recv got EOF from 127.0.0.1:40330 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 16:44:04.875223 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.867918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.31.61.189:40599)
0812 16:44:04.868115 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.868121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.868618 (+ 497us) server_negotiation.cc:407] Connection header received
0812 16:44:04.868897 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.868904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.869000 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.869075 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:04.869747 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.870926 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.872837 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.873555 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.873657 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.874355 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:04.874519 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:04.874579 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.874790 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.874877 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 16:44:04.886523 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:04.883868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.31.61.188:42817)
0812 16:44:04.884051 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.884057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.884665 (+ 608us) server_negotiation.cc:407] Connection header received
0812 16:44:04.884961 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.884967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.885083 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.885159 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.886273 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40336: BlockingRecv error: recv got EOF from 127.0.0.1:40336 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 16:44:04.902338 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.895151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.31.61.189:40599)
0812 16:44:04.895392 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.895405 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.896163 (+ 758us) server_negotiation.cc:407] Connection header received
0812 16:44:04.896341 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.896347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.896446 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.896555 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:04.896985 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.898130 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.900149 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.900860 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.900992 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.901530 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:04.901693 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:04.901784 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.901930 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.902022 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260812 16:44:04.918049 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:04.913556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.31.61.188:42817)
0812 16:44:04.913817 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.913824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.916209 (+ 2385us) server_negotiation.cc:407] Connection header received
0812 16:44:04.916470 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.916476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.916578 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.916688 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.917810 (+ 1122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40348: BlockingRecv error: recv got EOF from 127.0.0.1:40348 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
I20260812 16:44:04.938105 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.930557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.31.61.189:40599)
0812 16:44:04.930780 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.930791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.931356 (+ 565us) server_negotiation.cc:407] Connection header received
0812 16:44:04.931547 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.931557 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.931679 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.931796 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:04.932299 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.933497 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.935824 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.936548 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.936687 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.937239 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:04.937393 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:04.937489 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.937666 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.937767 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260812 16:44:04.956282 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:04.953915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.31.61.188:42817)
0812 16:44:04.954206 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.954212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.954317 (+ 105us) server_negotiation.cc:407] Connection header received
0812 16:44:04.954569 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.954573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.954668 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.954965 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.956064 (+ 1099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40350: BlockingRecv error: recv got EOF from 127.0.0.1:40350 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 16:44:04.983379 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:04.976613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.31.61.189:40599)
0812 16:44:04.977003 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.977011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.977193 (+ 182us) server_negotiation.cc:407] Connection header received
0812 16:44:04.977370 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.977376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.977483 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.977611 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:04.978121 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.979345 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.981169 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:04.981860 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:04.982033 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:04.982555 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:04.982706 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:04.982812 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:04.982964 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 16:44:04.983071 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
W20260812 16:44:04.993208 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:04.991010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.31.61.188:42817)
0812 16:44:04.991245 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:04.991252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:04.991516 (+ 264us) server_negotiation.cc:407] Connection header received
0812 16:44:04.991748 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:04.991753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:04.991883 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:04.992024 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:04.992938 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40352: BlockingRecv error: recv got EOF from 127.0.0.1:40352 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 16:44:05.034857 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:05.028956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.31.61.189:40599)
0812 16:44:05.029182 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:05.029189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:05.029463 (+ 274us) server_negotiation.cc:407] Connection header received
0812 16:44:05.029626 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:05.029632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:05.029749 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:05.029863 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:05.030313 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.031438 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.033055 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.033571 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.033713 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:05.034095 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:05.034212 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:05.034287 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:05.034451 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 16:44:05.034547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260812 16:44:05.046476 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:05.044234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.31.61.188:42817)
0812 16:44:05.044439 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:05.044445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:05.044879 (+ 434us) server_negotiation.cc:407] Connection header received
0812 16:44:05.045012 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:05.045017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:05.045125 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:05.045266 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:05.046262 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40360: BlockingRecv error: recv got EOF from 127.0.0.1:40360 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 16:44:05.120913 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:05.114735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.31.61.189:40599)
0812 16:44:05.114993 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:05.115000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:05.115278 (+ 278us) server_negotiation.cc:407] Connection header received
0812 16:44:05.115503 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:05.115508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:05.115580 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:05.115680 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:05.116100 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.116914 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.118775 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.119455 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.119729 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:05.120181 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:05.120321 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:05.120413 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:05.120538 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 16:44:05.120630 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 16:44:05.140265 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 16:44:05.133420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.31.61.188:42817)
0812 16:44:05.133597 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:05.133604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:05.138706 (+ 5102us) server_negotiation.cc:407] Connection header received
0812 16:44:05.138901 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:05.138906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:05.138994 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:05.139092 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 16:44:05.140000 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40368: BlockingRecv error: recv got EOF from 127.0.0.1:40368 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 16:44:05.228444 22936 catalog_manager.cc:1259] Loaded cluster ID: 04d053e390d349e48d2755613d9d67de
I20260812 16:44:05.228549 22936 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580: loading cluster ID for follower catalog manager: success
I20260812 16:44:05.231012 22936 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580: acquiring CA information for follower catalog manager: success
I20260812 16:44:05.232910 22936 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7852c1a2838e4032b596c131e4ffe580: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 16:44:05.277168 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:05.272191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.31.61.189:40599)
0812 16:44:05.272362 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:05.272366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:05.272761 (+ 395us) server_negotiation.cc:407] Connection header received
0812 16:44:05.272913 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:05.272918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:05.273021 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:05.273113 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:05.273523 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.274276 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.275675 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.276082 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.276176 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:05.276509 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:05.276612 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:05.276682 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:05.276815 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 16:44:05.276882 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 16:44:05.288552 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:05.282700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.31.61.188:42817)
0812 16:44:05.282870 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:05.282875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:05.283136 (+ 261us) server_negotiation.cc:407] Connection header received
0812 16:44:05.283270 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:05.283274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:05.283454 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:05.283572 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:05.284024 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.284761 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.286154 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.286596 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.286707 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:05.287254 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:05.287934 (+ 680us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:05.288053 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:05.288240 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 16:44:05.288319 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 16:44:05.306883 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:05.294407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.31.61.190:36085)
0812 16:44:05.300078 (+ 5671us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:05.300086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:05.300109 (+ 23us) server_negotiation.cc:407] Connection header received
0812 16:44:05.300247 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:05.300252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:05.300497 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:05.300785 (+ 288us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:05.301495 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.302477 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.304323 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.304719 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.304830 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:05.305473 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:05.305934 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:05.306053 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:05.306455 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 16:44:05.306552 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5518,"thread_start_us":5387,"threads_started":1}
I20260812 16:44:05.307097 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:05.294543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.31.61.188:42817)
0812 16:44:05.294738 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:05.294744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:05.300255 (+ 5511us) server_negotiation.cc:407] Connection header received
0812 16:44:05.300545 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:05.300551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:05.300761 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:05.300829 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:05.302098 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.302974 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.305017 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.305609 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.305725 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:05.306191 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:05.306290 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:05.306379 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:05.306700 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 16:44:05.306797 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 16:44:05.307349 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 16:44:05.300574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.31.61.189:40599)
0812 16:44:05.300798 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 16:44:05.300805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 16:44:05.301440 (+ 635us) server_negotiation.cc:407] Connection header received
0812 16:44:05.301611 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 16:44:05.301617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 16:44:05.301687 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 16:44:05.301760 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 16:44:05.302388 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.303416 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.305431 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 16:44:05.305941 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 16:44:05.306048 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 16:44:05.306604 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 16:44:05.306726 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 16:44:05.306806 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 16:44:05.307021 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 16:44:05.307105 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 16:44:05.312202 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22696
I20260812 16:44:05.313141 22761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:05.424970 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 22696
I20260812 16:44:05.443198 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22773
I20260812 16:44:05.444370 22837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:05.552409 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 22773
I20260812 16:44:05.569795 31990 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu pid 22849
I20260812 16:44:05.570986 22913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 16:44:05.678282 31990 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgT9LAJ/build/asan/bin/kudu with pid 22849
2026-08-12T16:44:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4327 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9686 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (368928 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST